ACS NSQIP Surgical Risk Calculator Is Not Affected by Studies Challenging Its Accuracy

Newswise — CHICAGO (April 4, 2017): The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator (SRC) was first released in 2013. The calculator was based on highly detailed and accurate NSQIP data collected from nearly 400 hospitals and 1.4 million patients and created a universal risk estimation tool that covered more than 1,500 unique surgical procedures across multiple subspecialties. The online tool allows surgeons to enter 23 preoperative patient risk factors about patients and then estimates risks for mortality and eight postoperative complications and lists these risks in comparison to “an average patient’s risks.” While recognized as an accurate, preeminent tool for estimating patients’ surgical risk, it is continually being expanded, refined, and evaluated; a 2016 update, relying on 2.7 million surgical records from nearly 600 hospitals, and 19 predictors covering nearly 1900 procedures, was also found to be accurate in internal studies.
However, in the last two years, 21 surgical research papers evaluated the use of the SRC and often concluded that the SRC was only a fair or poor predictor of outcomes following certain types of procedures. These studies were rigorously examined by researchers in the ACS Division of Research and Optimal Patient Care and surgical departments at the University of California Los Angeles and Washington University, St. Louis, Mo. Their analysis revealed that most of those 21 studies had important design limitations, such that findings should not disqualify the SRC for its intended goal—to provide a general purpose estimate of complication risk across a wide variety of operations, according to findings published online as an “article in press” on the Journal of the American College of Surgeons website in advance of print publication later this year.
The researchers conducted a two-part analysis. For each of the 21 studies, the first analysis reviewed the surgical focus, number of hospitals providing data, the largest number of events reported among the outcomes that were studied, and the metrics of accuracy that were applied. The second analysis used statistical simulations to assess how the previously determined study factors would affect estimates of risk calculator accuracy.
This analysis found that three aspects of study design, taken together, limited the reliability of conclusions that could be drawn with respect to the accuracy of the SRC in predicting surgical outcomes.
One factor was sample size. Only eight of the 21 studies involved more than 100 cases with events; yet unbiased and precise estimates of risk calculator performance metrics require larger sample sizes. “Statistical measures of discrimination and calibration are unreliable unless studies include 200 or more cases with events,” said Mark E. Cohen, PhD, one of the study’s authors and Statistical Manager in the ACS Division of Research and Optimal Patient Care.
Another factor was the homogeneity of case mix. The studies focused on small segments of very similar types of surgical procedures. A focus on narrow groups of operations, in and of itself, interferes with the ability of a statistical predictive model to discriminate among patients on the basis of risk. “The reduction in observed discrimination in these studies is likely due to the homogeneity of the populations. It does not reflect a problem in the way the SRC model has been specified,” Dr. Cohen added.
A third factor was the scope of the data sets. Most of the studies presented the experience of a single institution. “The calculator provides an estimate for the average NSQIP hospital but it is unlikely that any of these institutions represents the precisely average hospital. Thus, differences between observed rates and SRC predicted rates could be due to either the hospital not being average or the model not being accurate. These two causes cannot be disentangled in single-institution studies,” said Dr. Cohen.
“The risk calculator is a tool based on NSQIP, one of, if not the most, rigorous surgical data registries across many surgical disciplines. With data from the NSQIP program, the calculator can estimate the probability of the outcomes of a patient having an operation without a complication or the likelihood the patient will have a certain type of complication. It is intended to help further the patient/surgeon relationship and discussion of risk,” explained Clifford Y. Ko, MD, MS, MSHS, FACS, Director of the ACS Division of Research and Optimal Patient Care, which administers NSQIP.
“The calculator also helps surgeons improve the care of patients by identifying patients at high risk of a complication and planning interventions to mitigate that risk ahead of time,” said Dr. Ko, who is also a professor of surgery at UCLA David Geffen School of Medicine.
“The risk calculator is a useful tool in the armamentarium for providing high quality care to surgical patients,” Dr. Ko added.
“Surgeons should not be dissuaded from relying on the calculator for determining the risk for various outcomes,” said Dr. Cohen.
In addition to Drs. Cohen and Ko, other study authors include Yaoming Liu, PhD, and Bruce L. Hall, MD, PhD, MBA, FACS.
“FACS” designates that a surgeon is a Fellow of the American College of Surgeons.
Citation: An Examination of ACS NSQIP Surgical Risk Calculator Accuracy. Journal of American College of Surgeons.
# # #
About the American College of SurgeonsThe American College of Surgeons is a scientific and educational organization of surgeons that was founded in 1913 to raise the standards of surgical practice and improve the quality of care for all surgical patients. The College is dedicated to the ethical and competent practice of surgery. Its achievements have significantly influenced the course of scientific surgery in America and have established it as an important advocate for all surgical patients. The College has more than 80,000 members and is the largest organization of surgeons in the world. For more information, visit www.facs.org.