The translation of cancer research into real-world treatments is strikingly inefficient, with only 5% of potential cancer drugs making it from the research lab through phase III clinical testing, compared with the 20% of cardiovascular therapies that pass phase III testing. One potential explanation for this high failure rate is that very few cancer research findings are confirmed by independent studies. Recently, the major pharmaceutical company Amgen was only able to validate 6 out of 53—or 11% of—landmark cancer papers, but did not release the list of studies checked due to confidentiality issues. The Reproducibility Project: Cancer Biology was launched in an effort to build on this investigation in a more transparent manner.
In January, the Reproducibility Project released the results of its first five replication studies, one of which failed outright, while the other four yielded inconclusive or moderately successful results. In this perspective, John P.A. Ioannidis, MD, DSc, of Stanford University in Stanford, California—an expert on reproducibility—discusses how the failed replication study underscores the limitations of conceptual replication, which is the current standard for confirming biomedical research results.
Prior to the Reproducibility Project—which used exact replication—at least 10 other labs had seemingly validated the unreplicated paper’s findings that a particular peptide can enhance the efficacy of chemotherapy drugs. However, Ioannidis notes that these validation studies were conceptual replications instead of exact, which means their experimental conditions differed from those in the original study. For example, the validating labs looked at different cancer types or different anti-tumor peptides. Additionally, it is unknown whether these other labs tried multiple variants on the original study before finding one that yielded the desired results, since this information isn’t documented with conceptual replications.
Unfortunately, it is very difficult for researchers to publish exact replication studies because they aren’t novel, which is why conceptual replications proliferate. As this example from the Reproducibility Project shows though, encouraging and funding exact replication is essential to confirming cancer research results and accelerating the development of cancer treatments that will benefit patients.
“For those who question how much we can spend on replication, the answer is probably a thousand-fold more, easily,” said Ioannidis. “The entire [Reproducibility Project: Cancer Biology] costs $2 million. Waste in biomedical research is estimated to be tens, perhaps even hundreds of billions, annually. To do better, insights on reproducibility will be crucial. It is not about shaming and tarnishing reputations. It is about whether our observations are solid and eventually can have practical value.”
________________________________________
About AACCDedicated to achieving better health through laboratory medicine, AACC brings together more than 50,000 clinical laboratory professionals, physicians, research scientists, and business leaders from around the world focused on clinical chemistry, molecular diagnostics, mass spectrometry, translational medicine, lab management, and other areas of progressing laboratory science. Since 1948, AACC has worked to advance the common interests of the field, providing programs that advance scientific collaboration, knowledge, expertise, and innovation. For more information, visit www.aacc.org.
Clinical Chemistry is the leading international journal of clinical laboratory science, providing 2,000 pages per year of peer-reviewed papers that advance the science of the field. With an impact factor of 7.457, Clinical Chemistry covers everything from molecular diagnostics to laboratory management.
RSS