When an AI should stop and get a human

Vendors present escalation rate as a cost — a number to drive down, a share of conversations the software failed to close. For a clinic that framing is exactly backwards. The escalations are the conversations where a person was genuinely needed, and a system that escalates less is not better unless the knowledge base got better first.

What follows is how to design the boundary properly: what triggers a handoff, what the handoff carries, and what happens on the other side.

Topic first, confidence second

The common design escalates when the model’s confidence falls below a threshold. As the only mechanism, this is unsafe for clinical content, for a specific reason: a fluent, wrong, clinically-flavoured answer is a high-confidence output. Confidence catches the questions the system found hard. It does not catch the questions it found easy and should not have answered.

So the first gate is topic classification, applied before the model is asked to produce anything. Anything clinical routes to a human regardless of how confident the system is or how well the knowledge base covers it.

The softer signals

Below the hard rules sit signals that should escalate but are judgement calls rather than categories.

• Retrieval came back empty or weak. No source above threshold means no answer. Say so and hand over — this is the abstention case, and it should be logged as a knowledge gap rather than quietly retried.
• The patient repeated themselves. Two attempts at the same question means the first answer missed. A third attempt is a bad experience the system is manufacturing.
• Frustration in the language. Not sentiment scoring for its own sake — a patient who has become short is not going to be recovered by a better paragraph.
• The conversation has run long. Past a certain number of turns without resolution, the shape of the problem is not what the system thinks it is.
• Money beyond the published price. Refunds, disputes, custom packages, instalment arrangements. These are commercial decisions.
• A third party is messaging. A relative asking on behalf of a patient raises identity and consent questions worth a person’s attention.

What the handoff has to carry

An escalation that arrives as a bare notification — “conversation needs attention” — makes a staff member reconstruct everything. That reconstruction is the real cost of handoff, and it is entirely avoidable.

1. 01The trigger, stated plainly

   Not a code. “Patient described a symptom” or “no source found for question about laser hair removal on tattoos”. The staff member should know why they are reading this before they read anything else.

2. 02The full transcript

   Including anything the agent already said, so nobody repeats or contradicts it. This is where two systems answering the same patient causes visible damage.

3. 03The patient record

   History, previous treatments, upcoming appointments — pulled from the practice management system, not typed by the person picking it up.

4. 04Any attachment

   Photos and voice notes carried through as-is. Voice notes transcribed for skimming, with the original still playable.

5. 05A holding reply already sent

   The patient should not sit in silence while the queue moves. An honest message — that this is going to a person, and roughly when — costs nothing and prevents the patient messaging a competitor in the meantime.

The agent stands down

Once a human takes a conversation, the agent must stop replying in it. This sounds obvious and is routinely violated — a scheduled follow-up fires mid-conversation, an automation answers a message the nurse was already typing a reply to. The patient sees two voices, one of which contradicts the other.

The rule: handoff is a state on the conversation, not a notification. In the handed-off state the agent sends nothing at all, until a person explicitly hands it back.

Measure the boundary, not the automation rate

Three numbers describe whether the boundary is working, and none of them is “percentage automated”.

• Escalation precision — of conversations that escalated, how many genuinely needed a person? Chronically low precision means the topic rules are too broad and staff are being spammed into ignoring them.
• Escalation recall — of conversations that should have escalated, how many did? This is the number that matters for safety, and the only way to measure it is for someone to read a sample of non-escalated conversations every week.
• Escalation response time — how long a patient waits after handoff. Fast automation feeding a slow queue is a worse experience than a slow, consistent human one.