Who’s watching? Healthcare systems averaging 70 hidden AI applications, risking patient data

Healthcare organizations have unintentionally welcomed dozens of AI applications into their systems, potentially compromising sensitive patient data and established information hierarchies. What started with carefully selected AI implementations has evolved into a sprawling ecosystem of hidden AI capabilities embedded within everyday platforms like Microsoft Office, Salesforce, and Gmail. This silent AI proliferation creates significant governance challenges for healthcare institutions that must balance innovation with their strict regulatory obligations to protect patient information.

The big picture: Healthcare organizations are discovering 70 AI applications on average in their systems when security teams expected only 1-5, according to cybersecurity firm Prompt Security.

Why this matters: Patient data privacy is at risk when healthcare organizations lack visibility into how AI is being used within their existing software stack.

• These embedded AI capabilities can bypass traditional permission structures, potentially giving junior employees access to sensitive information they shouldn’t have.
• If confidential patient data is unknowingly shared with third-party language models, that information could be used to train these models, making it permanently accessible.

What they’re saying: “AI is growing at such a massive pace that this market is being fragmented, and AI is being integrated into any application,” explained Itamar Golan, CEO of Prompt Security.

• Golan described the moment healthcare organizations discover the true extent of AI in their systems as “a eureka moment.”
• “Once the information is embedded in the model’s brain, it’s a lost battle,” Golan warned about data leakage into large language models.

The hidden culprits: Common workplace applications have “quietly” embedded AI functionalities into their platforms without drawing attention.

• Microsoft Office, Adobe Acrobat, Bing, Salesforce, Gmail, Grammarly and LinkedIn are among the major applications now featuring built-in AI capabilities.
• Many healthcare leaders believe they’re controlling AI use by blocking ChatGPT or Gemini, while missing the AI already integrated into essential business applications.

The governance gap: Healthcare executives need visibility and policies specifically designed for this new reality of pervasive AI.

• Golan encourages continued AI adoption but emphasizes the need for “better visibility, to understand better which AI is already being adopted by whom, when, what data is being shared with it.”
• Only after gaining this comprehensive overview can organizations develop appropriate governance policies that protect patients while enabling innovation.