Researchers from Johns Hopkins University and the US Food and Drug Administration (FDA) have developed G-AUDIT, a tool to identify hidden issues in the datasets used to train medical AI.
Medical AI systems, while showing great potential for improving healthcare through disease detection, forecasting medical risks, opportunistic screening, and quantitative medicine, have repeatedly underperformed and exhibited serious implicit biases.
The researchers focused on biases resulting from the medical data on which AI systems are trained, which are often not representative of the larger population within which the AI systems are intended to be used.
"The models that drive precision medicine learn to infer clinical outcomes from the data they're trained on. In many cases, that works great, but it can also lead to interesting failures that aren't immediately apparent," said Mathias Unberath, an expert in AI-assisted medicine at Johns Hopkins University.
Data bias may result in AI shortcut learning, bias and fairness issues in AI predictions, and lack of generalization.
AI draws conclusions from patterns, relevant or irrelevant
Shortcut learning causes AI to draw conclusions from irrelevant patterns. In one example, an AI learned to recognize that skin markings made by clinicians were associated with malignant lesions. When tested, the AI produced 40% more false positives when the scan showed clinician markings.
Another AI was trained on a dataset including images from two different clinics, one which specialized in cancer and one generalized dermatology clinic. The clinics used separate imaging systems which varied in quality. Additionally, rulers were often present in the images from the cancer clinic as they are frequently used to track the growth of tumors. An AI can easily learn to associate camera quality and ruler presence with cancer risk.
"Envision taking this algorithm into the real world with a smartphone camera. There's no ruler. The camera quality is completely irrelevant. But the model learned to associate 'ruler' and 'camera.' You have just created health disparity," Unberath said.
"We train models to be the most predictive, but we have zero control over what the model uses to make the prediction," Unberath added.
These shortcuts mean that an AI may initially appear to be able to make diagnoses well, but will make crucial errors when applied outside the sample data.
G-AUDIT focuses on data sets themselves
While much of the research on medical AI biases focuses on adapting the AI models themselves, researchers from John Hopkins and the FDA focused on the data sets and developed a technique called Generalized Attribute Utility and Detectability-Induced Bias Testing (G-AUDIT).
G-AUDIT is designed to identify potential problems in data sets before they're used to train medical AI models. G-AUDIT was tested on a data set of images and on a set of health record texts used the characterize stigmatizing language usage by physicians.
Within the image data set, G-AUDIT found that image height, image width, and year of collection were at the highest risk for causing bias.
Within the text, G-AUDIT found that clinical specialty was at the highest risk of causing bias, more so than patient sex or race for several tasks.
The research team is planning to expand the tool and has said it may be adapted for use beyond healthcare.