If AI Scribes Were the Test Run, Supply Ontario Has Lessons to Learn

In May, Ontario’s Auditor General released a special report on the use of Artificial Intelligence (AI) in the Ontario government, including medical “AI scribes” that are increasingly being used by doctors, including us. While headlines and coverage focused mainly on so-called scribe hallucinations of these AI note-taking systems, that wasn’t the biggest news in the report.

The most significant finding is the overreliance of Supply Ontario, the provincial agency responsible for public procurement, on self-reported information from industry and vendors without independent verification, as well as the use of an evaluation criteria that undervalued data security.

The Auditor General noted the approval process placed relatively little emphasis on key safeguards, like security controls and bias mitigation. Significant gaps were also identified in vendor oversight. Less than half – nine of 20 approved vendors – provided independent third-party security audit reports. Instead, Supply Ontario evaluators largely relied on vendor self-attestation for data storage, privacy and security without independent verification. Supply Ontario also did not require vendors to conduct live demonstrations of their products, instead accepting simulated recordings to assess real-world performance and reliability.

Most recently, the Investigative Journalism Foundation identified one approved AI scribe vendor that used fake testimonials from nonexistent doctors.

Doctors in Ontario are not obligated to use these AI scribes. They can conduct their own due diligence, use another product or use no AI scribe at all. But this process matters because physicians were assured that approved AI scribes met “provincial requirements for clinical functions, privacy, and secure data storage.”

Don’t like ads? Become a supporter and enjoy The Good Men Project ad free

It seems those requirements were lacking.

The Auditor General’s report made a series of recommendations, including putting a greater emphasis on security, privacy and bias related to fairness, equity and language. The report recommended increasing vendor oversight through annual, independent assurance reports from all AI vendors, and requiring these reports as part of all software procurement submissions.

Independent verification, either via Supply Ontario or a third party, rather than industry self-report represents the most important Auditor General recommendation, one that is vital to build trust of both physicians who are using the tool, and patients whose data is captured.

Supply Ontario’s next major health-care project is leading procurement for a new, province-wide primary care electronic medical record (EMR) system, one with the potential to end fragmented, disconnected information sharing that frustrates both patients and providers. Done right, a patient’s medical record could follow them across clinics, reducing wasted time while clearing public and provincial standards for security and data governance. Supply Ontario’s oversight of the province-wide system could also halt the growing commercialization of primary care patient data, which largely benefits the pharmaceutical industry while putting patient privacy at risk.

While the lessons learned in the Auditor General’s report need to be implemented to build trust in AI scribe systems, the scribe procurement process was also a test run. The provincewide EMR is a far bigger project, with higher stakes. The lessons from the scribe process must be applied before EMR procurement begins, not after.

We write this as doctors who are users of these tools, and as researchers who study health technology. We are deeply invested in the role of technology in improving health care. To be sustained and lasting, this technology can only move at the speed of trust, and the ability of our public institutions to foster it.

—

Previously Published on healthydebate.ca with Creative Commons License

***

–