AI Models Used to Analyze Medical Images May Be Biased, Study Finds

A new study has revealed that AI models used to analyze medical images can exhibit biases in their diagnostic performance across different demographic groups, with the most accurate models showing the largest fairness gaps. The key findings and implications are:

AI models’ surprising capabilities linked to fairness gaps: The AI models that were most accurate at predicting patient demographics like race and gender from chest X-rays also had the biggest discrepancies in diagnostic accuracy between groups, suggesting the models may be relying on “demographic shortcuts” that lead to incorrect results for women, Black patients, and other groups.

• MIT researchers previously found AI models can predict a patient’s race from chest X-rays, something even skilled radiologists cannot do.
• Models with the highest accuracy in demographic prediction showed the largest “fairness gaps” – differences in diagnostic accuracy between races and genders.
• This implies the models use demographic information as a shortcut for making diagnoses, resulting in less accurate results for certain groups.

Debiasing methods have limitations: Techniques to reduce the models’ fairness gaps, such as optimizing for subgroup robustness or removing demographic information, worked well when models were tested on patients similar to their training data, but fairness gaps reemerged when applied to patients from different hospitals.

• Training models to optimize performance in the worst-performing subgroup or to disregard demographic data improved fairness when tested on patients from the same dataset.
• However, when these “debiased” models were tested on patients from other hospitals, large fairness gaps reappeared, despite overall accuracy remaining high.
• Group adversarial debiasing, which removes demographic information, maintained slightly better fairness on new datasets compared to subgroup robustness optimization.

Broader implications for medical AI deployment: The findings underscore the importance of hospitals thoroughly evaluating AI models on their own patient populations before use, as fairness guarantees from model developers may not generalize to new settings.

• AI diagnostic models are often trained on data from one hospital but deployed widely elsewhere.
• However, this study shows that fairness optimizations based on one patient dataset do not necessarily transfer to new patient populations.
• Hospitals should validate any external AI models on their own patients to ensure the models do not exhibit biases that could disadvantage certain groups.
• When possible, hospitals should train AI models on their own patient data for best results. Additional debiasing methods still need to be developed and tested.

In summary, this research reveals concerning links between medical image AI models’ ability to predict patient demographics and their tendency to give biased and inaccurate diagnoses for certain groups. Debiasing techniques have some benefits but do not fully generalize to new patient sets. Extensive on-site validation of models is critical before real-world use to protect against discriminatory algorithmic bias in medical AI systems.