AI chatbots show early signs of cognitive decline in dementia test

The limitations of AI language models are becoming even more apparent as researchers subject them to standardized cognitive tests typically used to assess human mental function.

Study overview: A recent BMJ study evaluated leading AI chatbots using the Montreal Cognitive Assessment (MoCA), a standard test for detecting early signs of dementia, revealing significant cognitive limitations.

• The study included major AI models including OpenAI’s GPT-4 and GPT-4o, Anthropic’s Claude 3.5 Sonnet, and Google’s Gemini 1.0 and 1.5
• GPT-4o achieved the highest score of 26 out of 30, barely meeting the threshold for normal cognitive function
• Google’s Gemini models performed particularly poorly, scoring just 16 out of 30

Key performance findings: The AI models demonstrated an uneven pattern of capabilities, excelling in some areas while showing marked deficiencies in others.

• The chatbots performed well in naming, attention, language, and abstraction tasks
• All models struggled significantly with visuospatial and executive functions
• The Gemini models failed basic delayed recall tasks, unable to remember a simple five-word sequence
• Researchers noted a concerning lack of empathy across all tested AI models, similar to symptoms of frontotemporal dementia

Medical implications: The study challenges widespread assumptions about AI’s readiness to take on significant medical responsibilities.

• The findings cast doubt on claims that AI will soon replace human doctors
• Cognitive limitations could affect the reliability of AI in medical diagnostics
• The lack of empathy and memory retention raises concerns about AI’s ability to process and respond to patient information effectively

Research context: While acknowledging the fundamental differences between human brains and large language models, the study deliberately evaluates AI on the tech industry’s own terms.

• The research responds to claims about AI consciousness and capabilities
• The approach highlights the gap between marketed capabilities and actual performance
• Results suggest AI models may be more limited than commonly portrayed

Looking ahead: The findings suggest a recalibration of expectations around AI in healthcare may be necessary, with AI potentially requiring its own form of cognitive care rather than being ready to provide it to humans. The irony of AI models potentially becoming “patients” rather than replacing medical professionals underscores the technology’s current limitations and the continued importance of human medical expertise.