Stanford’s new multimodal AI model predicts cancer treatment outcomes

Stanford researchers have developed a new AI model called MUSK that combines clinical notes and pathology images to predict cancer treatment outcomes and personalize patient care.

The innovation: MUSK (Multimodal transformer with Unified maSKed modeling) represents a significant advancement in medical AI by analyzing both clinical notes and pathology images without requiring manual data pairing.

• Unlike current AI models that rely on single data sources, MUSK mirrors how human pathologists make decisions by considering multiple types of medical information
• The model was pretrained on 50 million pathology images and 1 billion pathology-related text tokens covering 33 tumor types
• This large-scale training approach dramatically exceeds the data volume used in previous studies

Technical capabilities and testing: Initial testing of MUSK with data from over 8,000 patients has demonstrated promising results in several critical areas of cancer care.

• The model accurately predicts melanoma relapse and patient responses to immunotherapy for lung and gastro-esophageal cancers
• MUSK has shown strong performance in prognostic predictions across 16 cancer types, with particular success in breast, lung, and colorectal cancers
• The system’s ability to process unlabeled data eliminates the need for time-consuming manual labeling by human experts

Research context: The development was led by Stanford researchers and partially funded by the Stanford Institute for Human-Centered AI.

• Lead author Jinxi Xiang emphasized MUSK’s ability to extract complementary information from multiple data sources
• The research was published in Nature in early January 2025
• The project emerged from the lab of Associate Professor Ruijiang Li, which focuses on applying machine learning to medical imaging analysis and precision oncology

Future applications and limitations: While MUSK shows promise for revolutionizing cancer care, several steps remain before clinical implementation.

• The technology could potentially be adapted for other types of medical and biological data analysis
• Clinical trials and regulatory approval will be required before deployment in medical settings
• Researchers need to gather additional evidence to validate MUSK’s effectiveness

Looking ahead: While MUSK represents a significant advancement in multimodal foundation models for healthcare, its true impact will depend on successful clinical trials and regulatory approval. The model’s ability to process diverse, unpaired data sets could serve as a template for future medical AI systems that more closely mirror human diagnostic processes.