Does the Truth Wear Off?

Initial research findings sometimes diminish over time. Does this phenomenon apply to drug interactions?

In a recent article in The New Yorker Jonah Lehrer presents the startling thesis that results of scientific studies tend to fade over time in other words, the magnitude of findings in initial studies often diminish (and may even disappear) as more and more studies are done.

Lehrer proposes that this phenomenon is a general property of the scientific process, and may account for the fact that many scientific discoveries that create a big splash initially will eventually fade into obscurity. The question we will address in this article is whether this phenomenon applies to drug interactions as well. Lehrer presents several examples of this wearing off phenomenon from widely varying scientific fields: psychology, biology, drug therapy, and others. A classic example is the discovered by University of Washington graduate student Jonathan Schooler. He found that subjects shown a face were better able to identify the face later than subjects shown the face and then asked to verbally describe the face. Unfortunately, as Schooler and others have tried to replicate this effect, the magnitude has become smaller and smaller over time.For drug interactions, there are several situations in which initial reports are followed by a diminishing magnitude of effect, or sometimes even a total inability to replicate the original findings.Case Reports. Even with all we have learned about cytochrome P450 isozymes and adenosine triphosphate-binding cassette transporters over the past few decades, new drug interactions are still sometimes discovered through careful assessment of specific patient cases. It is not uncommon, however, to have 1 or more case reports suggest a drug interaction, followed by negative results in subsequent controlled studies. For example, there are case reports of warfarin interacting with almost every antibiotic available, yet when studied under controlled conditions, many of these interactions do not appear to occur (as we discussed in our Pharmacy Times column in June 2009).Thus, many case reports purporting to represent drug interactions turn out to be adverse effects from 1 of the drugs acting alone or are caused by some factor other than the addition of the second drug.Sample Size.Sometimes a drug interaction is initially studied using a small sample size, and due to the unusual sensitivity of 1 or 2 of the subjects the mean changes may be quite large. When the study is subsequently studied with adequate numbers of subjects, the magnitude of the effect may drop off substantially. When a small sample size includes people who are unusually resistant to the interaction, the study finds no interaction and is usually not accepted for publication, as described below.Publication Bias. Scientific journals naturally like to publish articles that are newsworthy, and studies finding no interaction between 2 drugs are generally not newsworthy. For example, if 5 different research groups study the same interaction between 2 drugs and only 1 finds an interaction, the positive study is the one most likely to be published. All of the studies may be scientifically valid, but only 1 found an interaction due to differences in study design (eg, type of subject, dose or duration of drug therapy, etc). Negative drug interaction studies are generally only newsworthy after an interaction has been accepted as real, and the negative study is attacking the accepted paradigm.

Perception Bias.This bias is a particular problem when the outcome measurement is subjective, such as looking for symptoms of serotonin syndrome following administration of 2 serotonergic drugs, or looking for adverse psychiatric effects of a drug combination. It is less of a problem when one is looking at plasma drug concentrations or laboratory results such as an international normalized ratio. Nonetheless, perception bias can result in initial drug interaction reports that are hard to replicate later. The wearing off effect, therefore, has been observed in drug interaction studies as well as other scientific fields. But the opposite of the wearing off phenomenon can also occur, in which initial reports suggest a small interaction, and subsequent study finds much larger effects. For example, initial wellcontrolled trials of the use of angiotensin-converting enzyme inhibitors and spironolactone in heart failure found an acceptably low risk of hyperkalemia. But after the combination was used extensively in actual practice by prescribers who did not select patients carefully and did not do appropriate monitoring, numerous cases of life-threatening and fatal hyperkalemia were reported.

Drs. Horn and Hansten are both professors of pharmacy at the University of Washington School of Pharmacy.

Reference 1. Lerher, J. The truth wears off: is there something wrong with the scientific method? The New Yorker.

December 13, 2010:52-57.