Deploying AI in patient messaging without creating a liability

Most compliance material about AI in healthcare is written at a level of abstraction that helps nobody — principles about transparency and fairness that cannot be checked on a Tuesday afternoon. This guide is the opposite. It is a list of controls you can implement, verify in your own system, and show to someone who asks.

Control 1: a hard clinical boundary, enforced before the model runs

The single most important control is a rule about what the system is permitted to answer, applied by topic classification before the language model is asked to produce anything. Not a prompt instruction. Not a confidence threshold. A routing decision.

The reason for the ordering is specific. A model instructed “do not give medical advice” will still do so when a question is phrased innocuously, because the instruction competes with the pressure to be helpful. A topic classifier that routes anything symptom-, medication-, diagnosis- or complication-shaped to a human queue removes the model from the decision entirely.

Write these categories down as clinic policy, not as vendor configuration. The vendor implements your policy; they do not author it. If a vendor cannot show you the categories in a readable form and let you change them, you do not control the boundary.

Control 2: grounding, so every answer has a provenance

Every reply the system sends should be traceable to a specific piece of approved source material. This is the control that makes the difference between “our assistant probably said something reasonable” and an answerable question.

Practically, this means retrieval over your own documents, with the retrieved passage stored alongside the sent message. When a patient disputes what they were told, you open the conversation and read the card. When a clinician spots a wrong answer, they fix the card and every future answer changes — which is also, incidentally, how the system improves.

• Every outbound message stores the retrieved source alongside it, permanently.
• Every knowledge card has a named owner and a review date, and clinical cards are approved by a clinician before use.
• Changing a card is logged with who changed it and when.
• When retrieval finds nothing above the confidence threshold, the system abstains rather than answering — and the abstention is logged as a knowledge gap.

Control 3: consent that is specific, recorded and revocable

Consent for messaging is not the same as consent for treatment, and consent for appointment reminders is not consent for marketing. Regimes across the region converge on the same expectations: purpose-specific, freely given, recorded, and as easy to withdraw as to give.

• State the channel by name. “We may contact you” is weaker than “we will send appointment reminders and aftercare messages by WhatsApp”.
• Separate clinical communications from promotional ones, with separate opt-ins.
• Record the wording shown, the date and the channel — not a boolean column that loses its meaning when the form is edited.
• Present it in Arabic and English, and treat consent given in either as equally valid.
• Honour withdrawal in any phrasing, in any language, on first request, and log it.

Disclosure that the patient is talking to an automated assistant belongs here too. It is required by several platform policies, expected by an increasing number of regulators, and — the practical argument — costs nothing. Patients who know they are talking to an assistant ask for a human when they want one, which is the outcome you want.

Control 4: data handling you can describe in one page

Four questions. If you cannot answer all four about your own deployment, that is the gap.

1. 01Where does the data physically sit?

   Name the region for the conversation store, the knowledge base and the backups. “The cloud” is not an answer. If your regulator or your hospital group requires in-region storage, this has to be contractual rather than a default setting.

2. 02Who can read a patient conversation?

   Enumerate the roles, including the vendor’s. Front desk, clinical staff and administrators should see different things. Vendor support access should require a documented reason and leave a trace.

3. 03How long is it kept?

   Set a retention window per data class and enforce it automatically. Conversations, attachments and transcripts may warrant different windows. Indefinite retention because nobody chose a number is the common state and the hardest to defend.

4. 04Is it used to train models?

   Get this in the contract, not in a support reply. The answer you want is that patient conversations are never used to train shared models, and that any vendor-side use for improving your own deployment is opt-in and describable.

Control 5: an audit trail that reconstructs the conversation

The test is a scenario. A patient complains, four months after the fact, that your clinic told them a treatment was safe for them. Can you produce: every message in that thread, in order; which were sent by software and which by a named member of staff; the source material behind each automated reply; the point at which it escalated and why; and who read it afterwards?

If the answer is yes, the AI in your messaging stack is an asset in that dispute rather than a liability — the record is more complete than any phone call would have produced. If the answer is no, the automation has removed accountability rather than adding capacity. There is no middle position here, and it is entirely determined by choices made at deployment.

The vendor questions worth asking in writing

• Can we see and edit the topic rules that decide what the agent will not answer?
• Show me a sent message and the exact source passage behind it, in the product interface.
• What does the system do when retrieval returns nothing relevant?
• Which region will our data live in, and is that contractual?
• Are our conversations ever used to train models, ours or anyone’s?
• Who at your company can read our patient conversations, under what process, and is it logged?
• What is the retention policy, and can we set it per data class?
• Can we export everything — conversations, knowledge base, audit log — in a readable format, on demand and at termination?
• What is your process when a customer reports that the agent said something wrong?
• What happens to conversations that escalate outside working hours?

The governance that keeps it working

Deployment controls decay. Three habits keep them alive, and none of them require a committee.

• A weekly escalation review. Someone clinical reads the week’s handoffs. Repeated escalations on the same informational topic mean a missing knowledge card; repeated escalations on clinical topics mean the boundary is working.
• A quarterly card review. Cards go stale silently. Prices, staff, protocols and product names change. Cards past their review date should stop being retrievable rather than continue answering.
• An incident path. When the agent says something wrong, there is a named person, a defined response, and a written record of what changed. Once a year, at minimum, you will need it.