The authors say that, in the course of assembling weekly lists of articles on obesity and nutrition, they began to notice more peer-reviewed articles containing what they refer to as ‘substantial or invalidating errors.’ “What was striking was how severe some of these errors were, involving mathematically impossible values, probabilities greater than one, weight loss results that, if true, would have required that adults had grown over 6 centimeters in height in two months, to name just a few,” said David B. Allison, Ph.D., leader of the research team and associate dean for Science in the UAB School of Public Health.
“These errors involved factual mistakes or practices which veered substantially from clearly accepted procedures in ways that, if corrected, might alter a paper’s conclusions,” said Andrew Brown, Ph.D., a scientist in the UAB School of Public Health and co-author of the commentary. “In several cases, our noting these errors led to retractions of the papers containing them.”
Brown says the team attempted to address more than 25 of these errors with letters to authors or journals. Their efforts revealed invalidating practices that occur repeatedly and showed how journals and authors react when faced with mistakes that need correction.
“We learned that post-publication peer review is not consistent, smooth or rapid,” Allison said. “Many journal editors and staff seemed unprepared to investigate, take action or even respond. Too often, the process spiraled through layers of ineffective emails among authors, editors and unidentified journal representatives, often without any public statement’s being added to the original article.”
During the informal 18-month review of literature, the authors found a number of recurring problems:
• Editors are often unprepared or reluctant to take speedy and appropriate action• Where to send expressions of concern is unclear• Journal staff who acknowledged invalidating errors were reluctant to issue retractions or even timely expressions of concern• Some journals may charge fees to authors who report the issues to correct others’ mistakes (more than $1,000)• No standard mechanism exists to request raw data for review to confirm the errors• Concerns expressed through online forums are easily overlooked and are not connected in a way to be found by readers of the article in question
The authors observed that there is little formal guidance for post-publication corrections. They recommend that journals should standardize their submission and peer-review processes, establish clear protocols to address expressions of concern, and waive publication fees associated with those expressions of concern.
Further suggestions include creating an environment to address readers’ concerns rapidly and provide clear information on how and to whom such concerns should be addressed.
“We also think it is very important to create an understanding that such expressions of concern are not a condemnation of the work, but should be viewed as an alert that the work is undergoing further scrutiny,” said co-author Kathryn A. Kaiser, Ph.D.
Additional recommendations suggest journals and statistical experts should work together to identify common statistical mistakes and that authors and journals should be prepared to share data and analysis code quickly when questions arise.
The authors noted common statistical errors in many of the studies, including mistaken design or analysis of cluster randomized trials, miscalculation in meta-analyses, and inappropriate baseline comparisons.
The authors acknowledge that their work did not constitute a formal survey and suggest that a more formal, systematic survey is needed to establish whether their experiences are representative of science in general.
“Ideally, anyone who detects a potential problem with a study will engage, whether by writing to authors and editors or by commenting online, and will do so in a collegial way,” Brown said. “Scientists who engage in post-publication review often do so out of a sense of duty to their community, but this important work does not come with the same prestige as other scientific endeavors.”
“Robust science needs robust corrections,” Allison added. “It is time to make the process less onerous.”
Co-authors of the commentary are Brown, Allison, Kaiser and Brandon J. George, Ph.D., of the UAB School of Public Health.
About UABKnown for its innovative and interdisciplinary approach to education at both the graduate and undergraduate levels, the University of Alabama at Birmingham is an internationally renowned research university and academic medical center, as well as Alabama’s largest employer, with some 23,000 employees, and has an annual economic impact exceeding $5 billion on the state. The five pillars of UAB’s mission include education, research, patient care, community service and economic development. UAB is a two-time recipient of the prestigious Center for Translational Science Award. Learn more at www.uab.edu. UAB: Knowledge that will change your world.
EDITOR’S NOTE: The University of Alabama at Birmingham is a separate, independent institution from the University of Alabama, which is located in Tuscaloosa. Please use University of Alabama at Birmingham on first reference and UAB on subsequent references.
VIDEO: www.youtube.com/uabnews TEXT: www.uab.edu/news TWEETS: www.twitter.com/uabnews
RSS