Online citations, reference lists, and bibliographies.

Toward Robust Information: Data Quality And Inter-rater Reliability In The American College Of Surgeons National Surgical Quality Improvement Program.

Mira Shiloach, Stanley K. Frencher, Janet E Steeger, Katherine S Rowell, Kristine Bartzokis, Majed G. Tomeh, Karen E. Richards, Clifford Y. Ko, Bruce Lee Hall
Published 2010 · Medicine
Cite This
Download PDF
Analyze on Scholarcy
Share
BACKGROUND Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure. STUDY DESIGN We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year. RESULTS Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables. CONCLUSIONS The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year.
This paper references
10.1002/bjs.1800810716
Introduction of a population-based vascular registry: validity of data and limitations of registration. The Finnvasc Study Group.
Mauri J Lepäntalo (1994)
10.1097/00000658-199810000-00006
The Department of Veterans Affairs' NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program.
Shukri F. Khuri (1998)
10.1093/intqhc/13.1.51
Completeness and accuracy of voluntary reporting to a national case registry of laparoscopic cholecystectomy.
Eva Dreisler (2001)
program specifics
ACS NSQI (2009)
10.1016/S0169-5002(09)70045-5
The National Lung Cancer Audit Database (LUCADA); essential analysis of data quality
Anna Rich (2009)
10.1016/S1072-7515(01)00940-1
Validating risk-adjusted surgical outcomes: site visit assessment of process and structure1
Jennifer Daley (1997)
10.1097/SLA.0b013e3181823485
Successful Implementation of the Department of Veterans Affairs’ National Surgical Quality Improvement Program in the Private Sector: The Patient Safety in Surgery Study
Shukri F. Khuri (2008)
10.1093/intqhc/8.2.153
Patients not included in medical audit have a worse outcome than those included.
J. Elfström (1996)
10.1097/00000658-200209000-00011
The National Surgical Quality Improvement Program in Non-Veterans Administration Hospitals: Initial Demonstration of Feasibility
A. Fink (2002)
10.1016/j.athoracsur.2003.07.030
Validity of the Society of Thoracic Surgeons National Adult Cardiac Surgery Database.
Karl F. Welke (2004)
10.1016/s1072-7515(97)00091-4
Validating risk-adjusted surgical outcomes: site visit assessment of process and structure. National VA Surgical Risk Study.
Jennifer Daley (1997)
The National Veterans Administration Surgical Risk Study: risk adjustment for the comparative assessment of the quality of surgical care.
Shukri F. Khuri (1995)
10.1016/S1072-7515(01)00938-3
Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study.
Shukri F. Khuri (1997)
participant use file user’s guide
ACS NSQI (2009)
10.1177/002205749203600413
The National.
(1892)
10.1016/s0003-4975(99)01538-6
The STS National database : Current changes and challenges for the new millennium. Commentary
Ferguson Tb (2000)
10.1016/S0003-4975(01)03339-2
A decade of change--risk profiles and outcomes for isolated coronary artery bypass grafting procedures, 1990-1999: a report from the STS National Database Committee and the Duke Clinical Research Institute. Society of Thoracic Surgeons.
Thomas Bruce Ferguson (2002)
Methodology to improve data quality from chart review in the managed care setting.
Laura D. Cassidy (2002)
10.1016/S0022-5347(01)63152-5
Risk adjustment of the postoperative morbidity rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study.
Jennifer Daley (1997)



This paper is referenced by
10.1016/j.jamcollsurg.2019.02.046
Association Between Hospital Safety Culture and Surgical Outcomes in a Statewide Surgical Quality Improvement Collaborative.
David Duston Odell (2019)
10.1016/j.surg.2018.07.030
Validation of day 1 drain fluid amylase level for prediction of clinically relevant fistula after distal pancreatectomy using the NSQIP database
Fady Daniel (2019)
10.1177/1945892418787129
Impact of Body Mass Index on Perioperative Outcomes of Endoscopic Pituitary Surgery
Yung-Jae Lee (2018)
10.1055/s-0033-1358382
Anesthesia duration as an independent risk factor for postoperative complications in free flap surgery: a review of 1,305 surgical cases.
Bobby D. Kim (2014)
10.3171/2013.10.PEDS13248
Return to system within 30 days of pediatric neurosurgery.
David M Wrubel (2014)
10.1016/j.spinee.2013.02.071
Patient factors, comorbidities, and surgical characteristics that increase mortality and complication risk after spinal arthrodesis: a prognostic study based on 5,887 patients.
Andrew J. Schoenfeld (2013)
10.13188/2332-4139.s200005
Validation of the ACS-NSQIP Liver Module: A Single-Center Experience
Aram N. Demirjian (2015)
10.1016/j.arthro.2019.06.021
Unplanned Admissions Following Hip Arthroscopy: Incidence and Risk Factors.
Jerry Y Du (2019)
10.1016/J.UROLOGY.2016.03.053
Re: Utilization of Preoperative Laboratory Testing for Low-Risk, Ambulatory Urologic Procedures.
David S. Wang (2016)
10.1016/j.arth.2016.05.031
Incidence, Risk Factors, and Sources of Sepsis Following Total Joint Arthroplasty.
Daniel D. Bohl (2016)
10.1016/j.jss.2016.08.082
Risk factors for superficial surgical site infection after elective rectal cancer resection: a multivariate analysis of 8880 patients from the American College of Surgeons National Surgical Quality Improvement Program database.
Elie Sutton (2017)
10.1016/J.JURO.2016.02.2086
PD02-12 EVALUATION OF UNPLANNED HOSPITAL READMISSIONS AFTER MAJOR UROLOGIC INPATIENT SURGERY IN THE ERA OF ACCOUNTABLE CARE
Benjamin V. Stone (2016)
10.5999/aps.2015.42.2.194
Early Surgical Site Infection Following Tissue Expander Breast Reconstruction with or without Acellular Dermal Matrix: National Benchmarking Using National Surgical Quality Improvement Program
Sebastian J Winocour (2015)
10.1016/j.arth.2014.07.004
Short term outcomes of revision total knee arthroplasty.
James Douglas Dieterich (2014)
10.1016/j.arthro.2017.04.019
Arthroscopic Versus Open Rotator Cuff Repair: Which Has a Better Complication and 30-Day Readmission Profile?
Dustin K. Baker (2017)
10.1016/j.jvs.2017.04.057
Thirty‐day outcomes after fenestrated endovascular repair are superior to open repair of abdominal aortic aneurysms involving visceral vessels
Prateek Ku Gupta (2017)
10.1016/j.arth.2017.11.064
Alternative Payment Models Should Risk-Adjust for Conversion Total Hip Arthroplasty: A Propensity Score-Matched Study.
Alexander S McLawhorn (2018)
10.1016/j.hpb.2018.07.006
Wound protectors mitigate superficial surgical site infections after pancreatoduodenectomy.
Jason B Liu (2019)
10.3171/2015.10.SPINE14582
Use of the modified frailty index to predict 30-day morbidity and mortality from spine surgery.
Rushna Ali (2016)
10.2106/JBJS.N.01016
The Effect of Smoking on Short-Term Complications Following Total Hip and Knee Arthroplasty.
Kyle R Duchman (2015)
10.1016/j.ijsu.2015.04.079
ASA class is a reliable independent predictor of medical complications and mortality following surgery.
Nicholas J. Hackett (2015)
10.1111/trf.13479
Variation in intraoperative and postoperative red blood cell transfusion in pediatric surgery.
A. Stey (2016)
10.1177/000313482008600325
Synoptic Operative Reporting: Documentation of Quality of Care Data for Rectal Cancer Surgery
Reagan L Robertson (2020)
10.1016/j.hpb.2018.01.008
Unique predictors and economic burden of superficial and deep/organ space surgical site infections following pancreatectomy.
Ayòtúndé B. Fadayomi (2018)
10.1002/lary.26848
Thirty-day morbidity and mortality following otologic/neurotologic surgery: Analysis of the national surgical quality improvement program.
Zachary G Schwam (2018)
10.1016/j.jor.2018.02.016
Postoperative complications in underweight patients undergoing total hip arthroplasty: A comparative analysis to normal weight patients.
Mikhail Zusmanovich (2018)
10.1016/j.jamcollsurg.2012.03.022
Perioperative risk of laparoscopic fundoplication: safer than previously reported-analysis of the American College of Surgeons National Surgical Quality Improvement Program 2005 to 2009.
Stefan Niebisch (2012)
10.14245/ns.1836008.004
Impact of Resident Involvement in Neurosurgery: An American College of Surgeons’ National Surgical Quality Improvement Program Database Analysis of 33,977 Patients
Andreea Seicean (2018)
10.1097/BRS.0000000000002747
Epidural Abscess: A Propensity Analysis of Surgical Treatment Strategies
Anisse N. Chaker (2018)
10.1007/s12630-013-9937-3
Morbidity and mortality after massive transfusion in patients undergoing non-cardiac surgery
Alparslan Turan (2013)
10.2106/JBJS.M.01490
Variations in data collection methods between national databases affect study results: a comparison of the nationwide inpatient sample and national surgical quality improvement program databases for lumbar spine fusion procedures.
Daniel D. Bohl (2014)
10.1016/j.juro.2016.12.001
Frailty and the Role of Obliterative versus Reconstructive Surgery for Pelvic Organ Prolapse: A National Study
Anne M Suskind (2017)
See more
Semantic Scholar Logo Some data provided by SemanticScholar