Addressing Disparities in AI-Driven Diagnostics: Ethical and Practical Implications

In a study evaluating the diagnostic accuracy of an online artificial intelligence (AI) application for various skin conditions, the investigators realized that there was a disparity between the accuracy for patients with darker skin tones than those with lighter skin tones, explained Scott D. Nelson, PharmD, MS, ACHIP, FAMIA, associate professor at Vanderbilt University Medical Center in Tennessee, during a presentation at the American Society of Health-System Pharmacists (ASHP) Pharmacy Futures 2024 in Portland, Oregon. Notably, the gap was found to be smaller with dermatologists due to their greater experience level in that space, but with primary care providers (PCPs), there was a much greater disparity, according to Nelson.

“With the PCPs, their diagnostic accuracy wasn't anywhere close to the dermatologists,” Nelson said during the ASHP presentation. “These researchers put together a diagnostic model, or a predictive model, for helping with diagnosis of skin conditions. What they saw was actually really cool. [The model] increased the accuracy of dermatologists by 33%, and with PCPs, [there was] a huge jump in their diagnostic accuracy, almost to about the level as the dermatologist without the AI tools.

However, when the investigators looked at the disparities between darker vs lighter skin tones, the disparities persisted with the dermatologists. Also, surprisingly, the use of the model increased the disparity gap with the PCPs.

Also, surprisingly, the use of the model increased the disparity gap with the PCPs. Image Credit: © AUNTYANN - stock.adobe.com

“There were more disparities with PCPs, even though their diagnostic accuracy was higher,” Nelson said. “So, this is an example of some of those ethical dilemmas that we face when evaluating these kinds of models. With the [value of] beneficence and respect for persons, the model improved accuracy, and accuracy is an important value that we have. We want to make sure that we're doing things accurately. [However, the model] also increased bias, and we also have a value of fairness and equity, so those 2 are in conflict in this example.”

Further, 2 values of efficiency and cost effectiveness are also in conflict in this case. Nelson noted that although the model may have decreased time to diagnosis, it also increased the cost of the diagnosis, because the model and the time it took to develop the model meant additional money was being spent for the diagnostic process.

“So, there's a lot of these conflicting values that we have to sort through and evaluate,” Nelson said. “Now this study is a pretty obvious example of racial disparities when we're looking at dermatology and skin tones. There was another study that was looking at imaging, so X rays, CT scans, ultrasound, and mammography in different body sites. The researchers trained the model to be able to predict self-reported race, which is kind of an interesting thing, even as to why they would want to do that, but it was used to illustrate the issue, I believe.”

Nelson explained further that this predictive model is able to predict self-reported race with an AUC of 0.9.

“So, it was very, very accurate,” Nelson said. “So, you could think, Well, there's probably slight differences in the structure, bone density, or something that's making it pick up on race. So, the researchers then corrupted the images by adding additional noise to them. They cropped the images to cut out certain parts of it and leave other parts in, and they did some other things, and still, the predictive model was able to predict self-reported race with an AUC gradient of 0.9.”

Nelson explained further that even the radiology experts, when looking at the images themselves, noted that they had no idea how the model would be able to predict self-reported race based on the images.

“So even with something that isn't as obvious as skin tones, [it] was still just a really interesting example of how biases could exist in the data even though we don't see them,” Nelson said. “There were some researchers that developed a model to assess the severity of illness for referring patients to some additional care, so some additional resources that are more expensive, but they wanted to identify those that could benefit the most from these resources, and it performed really well at predicting utilization.”

However, when the investigators looked at the fairness metrics, Black patients had 26% more chronic illnesses, according to Nelson. Black patients were also more likely to have uncontrolled hypertension and a higher systolic blood pressure.

“They had higher HbA1c values, and they were a lot sicker than their White counterparts, but they were assigned the exact same score in the predictive model,” Nelson said. “So, even outside of the imaging space, when we're just looking at other features or inputs into the model, these biases still existed. So that's something that is really important that we need to make sure that we're accounting for.”

Nelson noted that it’s important for health care professionals to understand that our health care data are biased.

“There are sex and gender differences. Some of this can be just because of the pathophysiology of the body, and we know a lot of this as pharmacists around the distribution of drugs being different between men and women, and how they respond to certain drugs, and the prevalence of different diseases across those populations,” Nelson said. “There's also differences in the data representation. So, heart disease is a leading cause of death in women, but 67% of participants in clinical studies are men, and so now we have this disparity in the data that is going into a predictive model, which is not actually representing the population that we're wanting to have a difference on. We see this a lot, too, with some minority groups that don't have as much access in interactions with the health care system, and so they’re underrepresented in the data, and we want to try to develop models that make sure that we include them.”

There are, however, some interesting cases where it may be okay to have some biased data, according to Nelson. When using a model for breast cancer, for example, it may be okay if that model performs better in women, and we can have other models or other tools for those known limitations or use cases, such as breast cancer in men.

“When we're looking at evaluating these models, we want to look at what are called the fairness metrics, which basically takes the performance of the model either in general or with a privileged class, whatever that class is, and then compares them with a subgroup to see if there is a significant difference in performance of the model,” Nelson said. “We have to think through these things and these ethical values, [around] improving outcomes for certain populations, but decreasing the outcomes for other populations, and we may be inadvertently perpetuating bias, because our health care data is biased.”

However, Nelson noted that there are things that can be done when designing the models to help decrease bias and result in less biased models than humans are capable of being.

“We all have our own subconscious biases, but those are some important factors to keep in mind when looking at these models, evaluating them, and considering the ethical considerations and dilemmas of using these models,” Nelson said.

Reference

Nelson SD. (Joseph A. Oddis Ethics Colloquium) Ethical Dimensions of AI in Pharmacy Practice, Part I: AI Behind the Counter. American Society of Health-System Pharmacists Pharmacy Futures 2024; June 8-12; Portland, Oregon.