Experts explain why AI models struggle to accurately diagnose cancer

AI’s latest attempts to diagnose cancer through pathology and imaging analysis demonstrate both promise and significant challenges in achieving clinical-grade accuracy.

Current landscape: The Mayo Clinic and Aignostics have developed Atlas, a new AI model trained on 1.2 million tissue samples, marking one of several recent efforts to apply artificial intelligence to cancer diagnosis.

• Atlas achieved 97.1% accuracy in identifying cancerous colorectal tissue, matching human pathologist diagnoses
• The model’s performance varied significantly across different cancer types, with only 70.5% accuracy for prostate cancer biopsies
• Overall, Atlas matched human expert diagnoses 84.6% of the time across nine benchmarks

Technical challenges: Processing pathology samples presents unique computational and data management hurdles that distinguish it from typical computer vision tasks.

• Tissue sample images contain over 14 billion pixels – 287,000 times larger than standard AI training images
• Researchers must use a “tile method” to break down these massive images into manageable pieces
• The process of selecting which image sections to analyze remains part art and part science

Data limitations: A critical bottleneck in developing effective AI diagnostic tools is the scarcity of digitized pathology data.

• Less than 10% of U.S. pathology practices have digitized their records
• The Mayo Clinic has undertaken a massive digitization effort, including 12 million archived slides
• Rare diseases pose a particular challenge, with some conditions having only 20 documented samples over a decade

Molecular analysis gap: The ability to predict molecular characteristics from tissue images remains a significant challenge.

• Atlas achieved only 44.9% accuracy in molecular testing benchmarks
• While this represents the best AI performance to date for molecular analysis, it falls far short of clinical requirements
• The performance gap highlights the complexity of inferring molecular-level information from visual tissue analysis

Clinical implementation hurdles: Getting AI models ready for real-world medical use requires extraordinary levels of accuracy.

• Experts suggest even 90% accuracy isn’t sufficient for clinical deployment
• Current models may be more suitable as assistive tools to help pathologists work more efficiently
• The lack of diverse training data limits AI models’ ability to identify rare or unusual cases

Looking ahead: While the current state of AI cancer diagnosis shows incremental progress, fundamental advances in model architecture and significantly larger datasets will be necessary to achieve meaningful clinical impact. The field remains optimistic but realistic about the considerable work ahead to reach truly reliable AI-assisted cancer diagnosis.