Poorly Designed Studies Lead to Overestimated Efficacy of Some Drugs

Researchers raise concerns that studies reporting negative results are not published, leading to overestimated drug efficacy.

According to new research from McGill University in Canada, poorly designed studies could lead to equally poor efficacy of drugs that waste money on trials that prove ineffective.

The findings underscore the importance of ensuring preclinical research can be reproduced by other scientists. The ability to reproduce an experiment helps confirm the credibility of the findings before clinical trials in humans take place, according to researchers.

“Only a fraction of drugs that show promise in animals end up proving safe and effective in humans,” said lead researcher Dr. Jonathan Kimmelman, who directs the Studies in Translation, Ethics and Medicine research group. “An important reason is because studies in animals are often not well designed, and because positive results have a higher chance of being published. They end up skewing what we think we know about the potential of a drug.”

Researchers analyzed all published animal studies on sunitinib, a cancer therapy successfully used to treat advanced kidney cancer.

Findings from the research indicated that studies reporting little or no anti-cancer effect were simply not published, leading anticancer effects of the drug to be overestimated by as much as 45%. However, these findings do not indicate any concern for clinical use of sunitinib.

Very little of the studies performed were blind or randomized, practices that help ensure personal expectations do not bias results. These methods are put in place so the researcher does not know which animal receives the control and which receives the drug.

Many times, it was unclear how many animals had been tested because the sample size was not reported.

The drug was tested against different types of cancer and all types of cancer tested showed a statistically significant anti-cancer response, a result that “strains credibility,” according to Kimmelman. In addition to overestimating the effects of the drug on cancer-prone mice, the studies also failed to observe the dose-dependent response to the drug that is known to occur in humans.

Lastly, the studies failed to test the drug on a variety of animal models, instead focusing on juvenile female mice with a compromised immune system. Other types of malignancies tested in mice that have spontaneously developed tumors showed less extreme effect sizes.

“Preclinical research is plagued by poor design and reporting practices, exposing patients to harmful and inactive agents, wasting time in the lab and driving up the price of drugs,” Kimmelman said.

This lack of rigor may explain why only 5% of agents that show an anti-cancer response in preclinical development are eventually licensed, versus the 20% rate in cardiovascular disease.

“Our findings provide compelling reasons for developing and implementing guidelines for the design and reporting of preclinical studies in cancer, similar to those already in use for stroke, epilepsy and cardiology,” Kimmelman said.