What should an AI agent be allowed to do in your clinic?
Motics Notes
Published

Motics Notes
Picture this: it is 9:25pm on Sunday and a new patient rings your clinic. Nobody is at the front desk, so the Phone Agent (as we call it) answers. It works out who the caller is, checks whether they have been seen before, finds a slot, books them in and sends a confirmation. By the time your receptionist arrives the next morning, the appointment is already in the diary.
That is a good outcome for the clinic and for the patient. It also raises a question most clinics have not yet asked their AI vendors: to do all of that, what was the agent allowed to see, engage with, or change within the clinic's existing software?
Integration used to mean moving data, now it means something more
For most of the history of clinic software, integration meant one thing: moving information from one system to another, so nobody had to re-type it. That is still valuable. But once an agent is acting on a clinic's behalf, with no human in the loop checking each step, integration becomes a question of what the agent is permitted to do inside the clinic's pre-existing systems.
Our co-founder and CEO, Dr Harvinder Power, describes the difference by comparing the Scribe with the Phone Agent. With the Scribe, a clinician reviews and signs off every clinical note, so a qualified person checks the output every time. The Phone Agent works on its own. As Harvinder puts it, "there's a lot of steps that happen without oversight. And so you need guardrails, safety measures and provenance built in, so you can understand and audit activities, even though a clinician may not have seen them take place."
The two agents need very different things from an integration, and a single "yes, we integrate" answer doesn't cover either issue.
Why open API access isn't safe for agents… yet
The quickest way to connect an agent to a practice management system is to give it access to that system's API and let it work out which actions to take. It is also the riskiest.
"If an agent can access an API, it will find a way to achieve its task using other elements it wasn't meant to," Harvinder says. An agent asked to book an appointment will look for whatever route completes the booking, and an open API offers many routes, including some that edit, overwrite or delete records the task never needed.
This has already gone wrong in the wider software industry, and Harvinder's view is that "it's only a matter of time until it comes to healthcare."
What going wrong would look like
In a clinic, the consequences are serious. And with the latest security risks with Google Cloud and PHP in the news, it is clearer than ever that vendors have to be mindful of security when patient data is on the line. The practice management system is at the heart of how a clinic runs, and an agent with broad access can act inside it. Harvinder is direct about the range of outcomes: "the smallest thing that could go wrong is a massive patient data breach," potentially millions of records. The worse outcome is a corrupted system of record, where "essentially every single interaction with that patient is now suspect," and the clinic has to go back through its records and contact its patients.
These are the risks a clinic takes on when an agent has open access to its records, and they are why the architecture around the agent matters as much as the agent itself.
Three principles we build to
At Motics integration is treated as an architecture decision. A decision we made at the start, this is guided by three core principles.
Agents do not take actions that cannot be undone
"We're essentially preventing things from being done that cannot be undone," Harvinder says, "such as, for example, booking somebody [in for] an operation. An agent today should not be doing that. In the future, things may change. But right now, with the way our clinical risk profile is, we need to be very careful about what we do with agents."
Every action is logged, attributable and auditable
A clinic should be able to see exactly what an agent did, when and why, without having to take the vendor's word for it.
Access is controlled at a deep level
"We control at a very deep level what agents can do," Harvinder says, "rather than just plug in APIs and say, great, an agent can just pick all the actions for you. That's unfortunately not safe."
Integration still matters
None of this is an argument against integration, quite the opposite, in fact. Patients have a smoother journey when the systems are connected, and clinic teams spend less time moving data around by hand. Motics Scribe integrates with Cliniko, Nookal, Meddbase and Semble for that reason.
But integration is not always how a clinic gets value from a product. The value itself comes from what the product does. "You're not buying it just for the sole reason that it's integrated," Harvinder says. "Because if that's the reason why you're buying it, the core experience can be awful, in which case the return on investment may never materialise. But great, it's integrated."
5 questions to ask any AI vendor
Before an AI agent goes near your patient records, we would ask these, in roughly this order.
What can the agent do in my practice management system: read data, write data, or both?
Which actions is it not allowed to take, and how is that enforced?
Can it take any action that cannot be undone?
Can I see a full log of every action it has taken, and who or what triggered it?
What does the product still deliver for my clinic if it is not integrated?
A vendor who can answer all five clearly has thought about safety. A vendor who answers with a list of integrations has answered a different question.
Clinics are right to ask whether a product integrates. As agents take on more of the work, it is worth asking what those agents are allowed to do as well.
Motics Notes is a series on how AI is changing the way private clinics operate. If you would like to talk through how this would work in your clinic, the Motics team is happy to help at motics.ai.