← Back to Search
GRADE Guidelines: 4. Rating The Quality Of Evidence--study Limitations (risk Of Bias).
G. Guyatt, A. Oxman, G. Vist, R. Kunz, J. Brozek, P. Alonso-Coello, V. Montori, E. Akl, B. Djulbegovic, Y. Falck-Ytter, S. Norris, J. Williams, D. Atkins, J. Meerpohl, H. Schünemann
Published 2011 · Medicine
Download PDFAnalyze on Scholarcy
In the GRADE approach, randomized trials start as high-quality evidence and observational studies as low-quality evidence, but both can be rated down if most of the relevant evidence comes from studies that suffer from a high risk of bias. Well-established limitations of randomized trials include failure to conceal allocation, failure to blind, loss to follow-up, and failure to appropriately consider the intention-to-treat principle. More recently recognized limitations include stopping early for apparent benefit and selective reporting of outcomes according to the results. Key limitations of observational studies include use of inappropriate controls and failure to adequately adjust for prognostic imbalance. Risk of bias may vary across outcomes (e.g., loss to follow-up may be far less for all-cause mortality than for quality of life), a consideration that many systematic reviews ignore. In deciding whether to rate down for risk of bias--whether for randomized trials or observational studies--authors should not take an approach that averages across studies. Rather, for any individual outcome, when there are some studies with a high risk, and some with a low risk of bias, they should consider including only the studies with a lower risk of bias.
This paper references
Randomized trials stopped early for benefit: a systematic review.
V. Montori (2005)
Prevention of venous thromboembolism: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition).
W. Geerts (2008)
When (not) to stop a clinical trial for benefit.
S. Pocock (2005)
Compression stockings for preventing deep vein thrombosis in airline passengers.
M. Clarke (2006)
Pharmacological therapy of spinal cord injury during the acute phase
J. Silver (2000)
Outcome reporting in industry-sponsored trials of gabapentin for off-label use.
S. Vedula (2009)
Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.
K. Schulz (1995)
Prevention of venous thromboembolism.
W. Geerts (2005)
The effect of beta‐blocker therapy on quality of life in heart failure patients: a systematic review and meta‐analysis
D. Dobre (2007)
Surgical interventions for lumbar disc prolapse.
J. N. Gibson (2007)
Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists.
D. Moher (1995)
Reporting bias and other biases affecting systematic reviews and meta-analyses: a methodological commentary
M. K. Sinha (2006)
Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors
A. Chan (2005)
during the acute phase
JM Brophy (2000)
Practical problems in interim analyses, with particular regard to estimation.
S. Pocock (1989)
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
A. Chan (2004)
Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography.
S. Sanderson (2007)
Low-molecular-weight heparins for thromboprophylaxis and treatment of venous thromboembolism in pregnancy: a systematic review of safety and efficacy.
I. Greer (2005)
Pharmacological therapy of spinal cord injury during the acute phase
V. Pointillart (2000)
-Blockers in Congestive Heart Failure: A Bayesian Meta-Analysis
J. Brophy (2001)
Comparison of registered and published primary outcomes in randomized controlled trials.
S. Mathieu (2009)
Methylprednisolone or naloxone treatment after acute spinal cord injury: 1-year follow-up data. Results of the second National Acute Spinal Cord Injury Study.
M. Bracken (1992)
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
A. Chan (2004)
Chapter 8: Assessing risk of bias in included studies
J. P. Higgins (2008)
Users' guides to the medical literature : a manual for evidence-based clinical practice
G. Guyatt (2002)
An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials.
J. Emerson (1990)
Beneficial effect of methylprednisolone sodium succinate in the treatment of acute spinal cord injury
K Otani (1994)
Systems to rate the strength of scientific evidence.
S. West (2002)
Increased cancer-related mortality for patients with type 2 diabetes who use sulfonylureas or insulin.
S. L. Bowker (2006)
Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
L. Wood (2008)
Testosterone use in men with sexual dysfunction: a systematic review and meta-analysis of randomized placebo-controlled trials.
Enrique R Boloña (2007)
Safety of low-molecular-weight heparin in pregnancy: a systematic review.
B. Sanson (1999)
Chloroquine or amodiaquine combined with sulfadoxine-pyrimethamine for treating uncomplicated malaria.
H. M. McIntosh (2005)
Association between unreported outcomes and effect size estimates in Cochrane meta-analyses.
T. Furukawa (2007)
An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods.
P. Devereaux (2004)
Evaluating non-randomised intervention studies.
J. Deeks (2003)
Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group
H. Soares (2004)
Jegar Pitchforth (2012)
Meta‐analysis of flavonoids for the treatment of haemorrhoids
P. Alonso-Coello (2006)
The hazards of scoring the quality of clinical trials for meta-analysis.
P. Jüni (1999)
Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis.
D. Bassler (2010)
[Pharmacological therapy of Cushing's disease].
I. Paiva (2005)
This paper is referenced by
Routledge handbook of physical activity and mental health
P. Ekkekakis (2013)
Somatostatin Analogues for the Treatment of Enterocutaneous Fistulas: A Systematic Review and Meta-analysis
S. Coughlin (2012)
Antibiotics at the time of removal of central venous catheter to reduce morbidity and mortality in newborn infants.
Rowena L McMullan (2018)
The cultural, organisational and contextual processes that might affect the implementation of massage in Lebanese neonatal intensive care units : a study informed by normalization process theory
B. Abdallah (2018)
Grading the strength of a body of evidence when assessing health care interventions: an EPC update.
N. Berkman (2015)
Gemcitabine-based chemotherapy for advanced biliary tract carcinomas.
O. Abdel-Rahman (2018)
Cataract surgery and age‐related macular degeneration. An evidence‐based update
L. Kessel (2015)
Basic study design influences the results of orthodontic clinical investigations.
Spyridon N. Papageorgiou (2015)
Pharmacotherapy for hypertension‐induced left ventricular hypertrophy
R. M. Finizola (2016)
Vitamin D supplementation in primary allergy prevention: Systematic review of randomized and non‐randomized studies
J. Yepes-Nuñez (2018)
Evaluating Published CER Evidence
J. Ren (2016)
Comparative effectiveness of the different components of care provided in heart failure clinics—protocol for a systematic review and network meta-analysis
Morgan Slater (2019)
Screening for chlamydia and/or gonorrhea in primary health care: protocol for systematic review
Jennifer Pillay (2018)
2017 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science With Treatment Recommendations Summary.
T. Olasveengen (2017)
Management of chest tubes after pulmonary resection: a systematic review and meta-analysis.
S. M. Coughlin (2012)
Nurse delivered lifestyle interventions in primary health care to treat chronic disease risk factors associated with obesity: a systematic review
G. Sargent (2012)
Health effects of water fluoridation
M. Sutton (2015)
Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study
S. O'Connor (2015)
Placebo, usual care and wait-list interventions for all mental health disorders
E. Faltinsen (2019)
Effectiveness and Safety of Glucocorticoids to Treat COVID-19: A Rapid Review and Meta-Analysis
S. Lu (2020)
Intravenous Acetaminophen Versus Placebo in Post-bariatric Surgery Multimodal Pain Management: a Meta-analysis of Randomized Controlled Trials
Y. Lee (2019)
Effectiveness of a 5-day summer handwriting camp for children ages 5 to 9 years old
Mackenzie Benning (2018)
DASH Dietary Pattern and Cardiometabolic Outcomes: An Umbrella Review of Systematic Reviews and Meta-Analyses
L. Chiavaroli (2019)
Serosorting and HIV/STI Infection among HIV-Negative MSM and Transgender People: A Systematic Review and Meta-Analysis to Inform WHO Guidelines
C. Kennedy (2013)
Effect of Task Load Interventions on Fatigue in Emergency Medical Services Personnel and Other Shift Workers: A Systematic Review
J. Studnek (2018)
The impact of vitamin D food fortification and health outcomes in children: a systematic review and meta-regression
Reem A Al Khalifah (2020)
Reply: Sperm retrieval rates by micro-TESE versus conventional TESE in men with non-obstructive azoospermia-the assumption of independence in effects sizes might lead to misleading conclusions.
G. Corona (2020)
Systematic Review and Meta-Analysis of Pigtail Catheter Use in Traumatic Pneumothorax and Hemothorax
Anthony J. Carden (2017)
Understanding the nature and scope of clinical research commentaries in PubMed
James R. Rogers (2020)
A Systematic Review of Nonpharmacological Interventions for Moderate to Severe Dementia: A Study Protocol for a Systematic Review and Meta-Analysis
Riyoung Na (2018)
Needle aspiration versus intercostal tube drainage for pneumothorax in the newborn.
M. Bruschettini (2016)
Critical reading and critical thinking--study design and methodology: a personal approach on how to read the clinical literature.
T. Lipman (2013)See more