Navigating the Deluge of Clinical Literature During and After COVID-19: Insights From ASHP Pharmacy Futures 2024

There has been exponential growth in published clinical literature over the past few years, with the COVID-19 pandemic acting as a catalyst for this growth, explained presenter Emily K. Frederick, PharmD, BCPS, associate professor and director of student affairs at Sullivan University College of Pharmacy and Health Sciences, during a session at the American Society of Health-System Pharmacists (ASHP) Pharmacy Futures 2024 in Portland, Oregon. This growth in published clinical literature was particularly present during the COVID-19 pandemic, according to Frederick.¹

Co-presenter Elyse A. MacDonald, PharmD, MS, BCPS, FASHP, director of pharmacy, Investigational Drug Service, Stanford Health Care, explained that during the first 10 months of the COVID-19 pandemic, there were more than 125,000 articles published on COVID-19 pandemic–related information. Then, in the first 19 months of the pandemic, 154,564 articles were published on COVID-19–related information.¹

“I have a hard time reading 5 articles at a time, so the fact that there's over 154,000 articles, it’s just overwhelming to me,” MacDonald said during the ASHP session. “It [feels] like, ‘Well, I think I need to look at everything.’ And, in the end, you just get anxiety [about the number of articles] because you want to do the best for your patients, and you get that feeling that you have to go through as much as you can, and it's just not possible.”¹

By the time of the ASHP Pharmacy Futures 2024 presentation in June, MacDonald noted that the World Health Organization had made available more than 500,000 articles in their research database on COVID-19–related information.¹

“So, even now, it's just getting worse,” MacDonald said. “[During the pandemic,] the information was everywhere. It was not just in our peer reviewed journals, but it was on social media, Twitter, the nightly news—it was just so much.”¹

MacDonald noted that if she prioritized attending online meetings, CME events, and webinars during the pandemic on COVID-19–related information, the information shared in those meetings could become outdated by the next day.¹

“It's like, what do we do with that,” MacDonald said. “How are we to keep up with it?”¹

Additionally, MacDonald noted that with the proliferation of misinformation during the COVID-19 pandemic about the SARS‑CoV‑2 virus and how it could be spread, it was difficult to discern the right recommendation for a patient at any given time.¹

With the proliferation of misinformation during the COVID-19 pandemic about the SARS‑CoV‑2 virus and how it could be spread, it was difficult to discern the right recommendation for a patient at any given time. Image Credit: © Mongkolchon - stock.adobe.com

“You think, ‘Oh, my gosh, the recommendation that I might have made, while it was the best at that time, thinking back on it—hopefully I didn't harm my patient,’” MacDonald said. “It's like this hurricane or tornado coming at you and you just don't really know what to do… There was just so much misinformation during COVID.”¹

By the end of December 2020, there were 72 retractions of articles discussing COVID-19 treatments and care, MacDonald explained.^1,2 By May 2022, there were 138 retractions of published literature, and by June 2024, there were 421 retractions.^1,2 For comparison in the field of infectious diseases, MacDonald explained that Ebola has had 2467 articles published with none retracted, H1N1 has had 4689 articles published with 2 retractions, SARS has had 1884 articles published with no retractions, and Zika has had 1968 articles published with no retractions.¹

“Where we get this information from is called Retraction Watch. They have a special COVID-19 section that you can go in and view all the articles that were retracted,” MacDonald said. “It's a really great example of what went on during COVID-19, because if I heard one more thing about hydroxychloroquine or ivermectin, it was like, ‘Oh my gosh, just stop.’”¹

Yet, when looking at the rate of retractions in relation to the amount of literature published on the virus, the retraction rate for COVID-19 was similar to that of H1N1.¹

“The retraction rate for COVID-19 papers was about 0.3%, with H1N1 having an overall retraction rate of 0.4%. So, there was actually virtually not that much of a difference [in] the COVID-19 misinformation,” MacDonald said. “We knew it existed, but the overall rate and percent of retraction really isn't different from what we would normally see.”¹

When looking at shared characteristics of retracted articles during the COVID-19 pandemic, MacDonald noted that authors of an article published in JAMA Network Open assessed the impact factor, time to retraction, and number of citations for these articles.^1,2 Specifically, between February 1, 2020, and May 5, 2022, the investigators observed that 138 articles were retracted on COVID-19–related information, with approximately 82% retracted within 6 months, and approximately 61% retracted with no reason provided. For non–COVID-19 related articles, there were 380 retractions, with approximately 60% retracted within 6 months, and approximately 33% retracted with no reason.^1,2

“Sometimes the [journal] will publish the reason for the retraction,” MacDonald said. “There could be research misconduct, or things like that. But two-thirds of the time for the COVID-19 articles, they just didn't tell us why [they were retracted].”¹

These data also show that there is a difference in the number of COVID-19–related articles retracted in a 6-month time frame, according to MacDonald.¹

“I think it just goes to show that there's just so much information coming out with COVID-19,” MacDonald said. “It was just rapidly changing so much that the articles needed to be retracted, and sometimes we just didn't know it [at the time].”¹

For impact factor of the retracted articles, the investigators assessed the number of citations the retracted articles had in other publications. According to MacDonald, the median impact factor of the retracted articles on COVID-19–related information was 3.13.¹

“I think the thought is that the higher the impact factor was, the less retracted articles [there would be],” MacDonald said. “[But,] I don't think that's necessarily true, given that the median impact factor is 3.13 for the articles that were retracted.”¹

Additionally, the median time to retraction for these articles was about 120 days, and the median number of citations per article was 6, according to MacDonald.¹

“So, quite a few citations there,” MacDonald said. “Also, this doesn't mean that articles that will be retracted will fall in line with [these numbers], but it’s just interesting information as you might think that articles in 1 journal might be retracted more than another.”¹

With all of the information that was published at a rapid pace during the COVID-19 pandemic, MacDonald noted that questions have since arisen regarding the role of artificial intelligence (AI) in helping clinicians process some of this information more quickly, according to MacDonald. She explained that at ASHP Midyear 2023, there were 2 studies presented that assessed responses from ChatGPT to questions regarding drug information.¹

“That was really interesting to me, because my training is in drug information,” MacDonald said. “Colleagues at the Long Island University’s College of Pharmacy Drug Information Service thought, ‘Hey, let's see how the free version of ChatGPT answers questions about drug information, and let's see how accurate they are.’ So, they had a list of 39 questions that they asked ChatGPT, and they also had answered these questions themselves in advance.”¹

Based on the results of asking the free version of ChatGPT these questions, the investigators found that 10 out of the 39 responses were satisfactory.^1,3 However, 29 out of the 39 answers were not deemed satisfactory by the pharmacists.¹

“So, they evaluated it a little further, and [found that] 11 out of those 29 answers did not address the question that was asked, so they got an irrelevant answer,” MacDonald said. “Then an additional 10 had incomplete or inaccurate response. So maybe some of it was right, but it didn't really answer the full question.”¹

For each question, the investigators also asked ChatGPT to list the references used for the responses provided. However, ChatGPT was only able to provide references for 8 of the 39 answers given.¹

“Basically, the references for most of the answers included references that didn't exist at all. So, if it listed a journal article, and you would go to look for that journal article, it just didn't exist. So, I guess [ChatGPT] just made it up,” MacDonald said. “I thought that was a really interesting takeaway, and something that we need to keep in mind.”¹

Another study was presented at ASHP Midyear 2023 on this subject by pharmacists at the Creighton University Center for Drug Information and Evidence-Based Practice, according to MacDonald. The group there decided to look at what they called “low-level drug information questions” and “high-level drug information questions.”¹

“So, what they did is they submitted questions from their database to ChatGPT to see what kind of responses they got,” MacDonald said. “These low-level questions ranged from just general drug data, drug information questions, or questions that really didn't require analysis or clinical thinking and clinical judgment. And basically, you could answer them with tertiary literature. So largely textbooks, lexicons, or something like that.”¹

The investigators placed questions within various categories, such as adverse drug reactions, contraindications, warnings, precautions, label indications, pharmacokinetics, and dosing and administration information, according to MacDonald.¹

“They looked at the accuracy of the answer from ChatGPT, and they also looked at whether there was a certain type of question that was easier for ChatGPT to answer,” MacDonald said. “What they found was that about 75% of the responses [for low-level drug information questions] were accurate and 12% were inaccurate, which is not too bad for these, in my opinion.”¹

As far as the type of question, pharmacokinetics and label indications were approximately 90% accurate, MacDonald explained.¹

“I think, again, that's a really good return rate,” MacDonald said. “Then, for contraindications, warnings, and precautions, about 60% of the time, ChatGPT provided accurate information.”¹

For high-level drug information questions, the investigators posed questions that required more clinical judgment or did not necessarily have a binary answer.¹

“They submitted 10 high-level drug information questions to ChatGPT, and ChatGPT provided 27 citations with answers for those questions. Similarly to the colleagues at the Long Island University’s College of Pharmacy Drug Information Service, these high-level questions had fake references [provided] almost 80% of the time,” MacDonald said. “Also, about 15% of the time, the references didn't make sense.”¹

MacDonald noted that since this analysis was conducted in the spring of 2023, the investigators assessed how much ChatGPT had evolved in its responses by August 2023.¹

“In August, they submitted these same high-level questions, and asked for references again. [This time,] ChatGPT didn't list any references, and instead basically said to consult your health care provider,” MacDonald said. “You can see the difference there in the references. I found that quite interesting that ChatGPT caught on and was like, ‘Oh, okay, we maybe shouldn't be spitting out fake references where no one can find the journal article.’ So, I think it's very eye opening that some of the concerns that we have with AI have been proven true here.”¹

With these problems with accuracy of responses and the use of fake references, the potential for harm was demonstrated to be significant if consumers were to use ChatGPT to obtain drug information relevant to their health care or related health care decisions.¹

“The Federal Trade Commission [FTC] is investigating OpenAI for consumer harm,” MacDonald said. “They're asking for descriptions on how the program will be tweaked and manipulated to improve the algorithm and produce different results. Then [FTC] also wants these systems or programs to describe how they're going to deal with hallucinations, which is an industry term for basically false information [provided by AI].”¹

According to MacDonald, these concerns tie back to what was found in the findings of the investigators at Long Island University and at Creighton University presented at ASHP Midyear 2023 in December.¹

“It’s interesting that ChatGPT has changed, and now, for those types of medical or clinical questions, [ChatGPT takes an approach of,] ‘We're not responsible, you need to talk to your health care provider,’ which I think is what we want,” MacDonald said.¹

REFERENCES

1. Frederick EK, MacDonald EA. Clinical Currents and AI Horizons: Surfing the Waves of Clinical Trends and Literature. American Society of Health-System Pharmacists Pharmacy Futures 2024; June 8-12; Portland, Oregon.

2. Shi X, Abritis A, Patel RP, et al. Characteristics of Retracted Research Articles About COVID-19 vs Other Topics. JAMA Netw Open. 2022;5(10):e2234585. doi:10.1001/jamanetworkopen.2022.34585

3. Constantino AK. Free ChatGPT may incorrectly answer drug questions, study says. CNBC. December 5, 2023. Accessed June 12, 2024. https://www.cnbc.com/2023/12/05/free-chatgpt-may-incorrectly-answer-drug-questions-study-says.html