Hospitals continue to adopt AI transcription tools despite OpenAI warnings

AI transcription tools in healthcare: Risks and widespread adoption: Despite warnings from OpenAI, hospitals and healthcare providers are increasingly adopting error-prone AI transcription tools, raising concerns about patient safety and data integrity.

• An Associated Press investigation has uncovered the widespread use of OpenAI’s Whisper transcription tool in medical and business settings, despite its tendency to generate fabricated text.
• Over 12 experts have confirmed that the model frequently invents text that was never spoken, a phenomenon known as “confabulation” or “hallucination” in AI circles.
• A University of Michigan researcher found that Whisper created false text in 80% of public meeting transcripts examined, while another developer discovered invented content in almost all of 26,000 test transcriptions.

OpenAI’s warnings and healthcare industry response: OpenAI has explicitly cautioned against using Whisper for “high-risk domains,” yet the healthcare industry has largely ignored these warnings in favor of potential cost savings and efficiency gains.

• More than 30,000 medical workers are currently using Whisper-based tools to transcribe patient visits, highlighting the rapid adoption of this technology in healthcare settings.
• Forty health systems have implemented a Whisper-powered AI copilot service from Nabla, which acknowledges Whisper’s potential for confabulation but erases original audio recordings, potentially compromising the ability to verify transcription accuracy.

Potential consequences and ethical concerns: The use of AI transcription tools that generate false information raises significant ethical and practical concerns, particularly in healthcare settings where accuracy is crucial.

• Researchers from Cornell and the University of Virginia found Whisper adding non-existent violent content and racial commentary to neutral speech in 1% of samples, illustrating the potential for serious misrepresentations in medical records.
• The erasure of original audio recordings by some service providers further complicates the issue, making it difficult to verify the accuracy of AI-generated transcriptions and potentially compromising patient care and legal proceedings.

Technical underpinnings of Whisper’s confabulation: Understanding the reasons behind Whisper’s tendency to generate false information is crucial for addressing the issue and developing more reliable AI transcription tools.

• Whisper’s confabulation is related to its prediction-based design and the nature of its training data, which includes 680,000 hours of web-sourced audio.
• The inclusion of YouTube videos and other online content in the training data has likely contributed to issues with overfitting and inappropriate outputs, highlighting the importance of carefully curating AI training datasets.

Industry motivations and regulatory implications: The healthcare industry’s rapid adoption of AI transcription tools, despite known risks, underscores the need for more robust regulation and oversight in this area.

• Healthcare companies are driven to adopt “good enough” AI tools to cut costs and improve efficiency, even when these tools may compromise data accuracy and patient safety.
• The author suggests that regulation and certification may be necessary for AI tools used in medical settings to ensure their reliability and protect patient interests.

Broader implications for AI in healthcare: The widespread adoption of potentially unreliable AI transcription tools in healthcare settings raises important questions about the broader implications of AI integration in medicine.

• This case highlights the tension between technological innovation and patient safety, emphasizing the need for careful evaluation and testing of AI tools before their deployment in critical healthcare applications.
• As AI continues to play an increasingly significant role in healthcare, striking a balance between innovation and risk mitigation will be crucial for maintaining public trust and ensuring the highest standards of patient care.