Online citations, reference lists, and bibliographies.
← Back to Search

GRADE Guidelines: 4. Rating The Quality Of Evidence--study Limitations (risk Of Bias).

G. Guyatt, A. Oxman, G. Vist, R. Kunz, J. Brozek, P. Alonso-Coello, V. Montori, E. Akl, B. Djulbegovic, Y. Falck-Ytter, S. Norris, J. Williams, D. Atkins, J. Meerpohl, H. Schünemann
Published 2011 · Medicine

Cite This
Download PDF
Analyze on Scholarcy
Share
In the GRADE approach, randomized trials start as high-quality evidence and observational studies as low-quality evidence, but both can be rated down if most of the relevant evidence comes from studies that suffer from a high risk of bias. Well-established limitations of randomized trials include failure to conceal allocation, failure to blind, loss to follow-up, and failure to appropriately consider the intention-to-treat principle. More recently recognized limitations include stopping early for apparent benefit and selective reporting of outcomes according to the results. Key limitations of observational studies include use of inappropriate controls and failure to adequately adjust for prognostic imbalance. Risk of bias may vary across outcomes (e.g., loss to follow-up may be far less for all-cause mortality than for quality of life), a consideration that many systematic reviews ignore. In deciding whether to rate down for risk of bias--whether for randomized trials or observational studies--authors should not take an approach that averages across studies. Rather, for any individual outcome, when there are some studies with a high risk, and some with a low risk of bias, they should consider including only the studies with a lower risk of bias.
This paper references
10.1001/JAMA.294.17.2203
Randomized trials stopped early for benefit: a systematic review.
V. Montori (2005)
10.1378/chest.08-0656
Prevention of venous thromboembolism: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition).
W. Geerts (2008)
10.1001/JAMA.294.17.2228
When (not) to stop a clinical trial for benefit.
S. Pocock (2005)
10.1002/14651858.CD004002.PUB3
Compression stockings for preventing deep vein thrombosis in airline passengers.
M. Clarke (2006)
10.1038/SJ.SC.3101007
Pharmacological therapy of spinal cord injury during the acute phase
J. Silver (2000)
10.1056/NEJMsa0906126
Outcome reporting in industry-sponsored trials of gabapentin for off-label use.
S. Vedula (2009)
10.1001/JAMA.273.5.408
Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.
K. Schulz (1995)
10.1378/chest.128.5.3775
Prevention of venous thromboembolism.
W. Geerts (2005)
10.1002/PDS.1234
The effect of beta‐blocker therapy on quality of life in heart failure patients: a systematic review and meta‐analysis
D. Dobre (2007)
10.1097/brs.0b013e3180bc2431
Surgical interventions for lumbar disc prolapse.
J. N. Gibson (2007)
10.1016/0197-2456(94)00031-W
Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists.
D. Moher (1995)
10.1586/14737167.6.5.603
Reporting bias and other biases affecting systematic reviews and meta-analyses: a methodological commentary
M. K. Sinha (2006)
10.1136/bmj.38356.424606.8F
Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors
A. Chan (2005)
during the acute phase
JM Brophy (2000)
10.1016/0197-2456(89)90059-7
Practical problems in interim analyses, with particular regard to estimation.
S. Pocock (1989)
10.1503/cmaj.1041086
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
A. Chan (2004)
10.1093/IJE/DYM018
Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography.
S. Sanderson (2007)
10.1182/BLOOD-2005-02-0626
Low-molecular-weight heparins for thromboprophylaxis and treatment of venous thromboembolism in pregnancy: a systematic review of safety and efficacy.
I. Greer (2005)
10.1038/sj.sc.3100962
Pharmacological therapy of spinal cord injury during the acute phase
V. Pointillart (2000)
10.7326/0003-4819-134-7-200104030-00008
-Blockers in Congestive Heart Failure: A Bayesian Meta-Analysis
J. Brophy (2001)
10.1001/jama.2009.1242
Comparison of registered and published primary outcomes in randomized controlled trials.
S. Mathieu (2009)
10.3171/JNS.1992.76.1.0023
Methylprednisolone or naloxone treatment after acute spinal cord injury: 1-year follow-up data. Results of the second National Acute Spinal Cord Injury Study.
M. Bracken (1992)
10.1001/JAMA.291.20.2457
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
A. Chan (2004)
Chapter 8: Assessing risk of bias in included studies
J. P. Higgins (2008)
Users' guides to the medical literature : a manual for evidence-based clinical practice
G. Guyatt (2002)
10.1016/0197-2456(90)90175-2
An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials.
J. Emerson (1990)
Beneficial effect of methylprednisolone sodium succinate in the treatment of acute spinal cord injury
K Otani (1994)
Systems to rate the strength of scientific evidence.
S. West (2002)
10.2337/DIACARE.29.02.06.DC05-1558
Increased cancer-related mortality for patients with type 2 diabetes who use sulfonylureas or insulin.
S. L. Bowker (2006)
10.1136/bmj.39465.451748.AD
Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study
L. Wood (2008)
10.4065/82.1.20
Testosterone use in men with sexual dysfunction: a systematic review and meta-analysis of randomized placebo-controlled trials.
Enrique R Boloña (2007)
10.1055/S-0037-1614550
Safety of low-molecular-weight heparin in pregnancy: a systematic review.
B. Sanson (1999)
10.1002/14651858.cd000386
Chloroquine or amodiaquine combined with sulfadoxine-pyrimethamine for treating uncomplicated malaria.
H. M. McIntosh (2005)
10.1001/JAMA.297.5.468-B
Association between unreported outcomes and effect size estimates in Cochrane meta-analyses.
T. Furukawa (2007)
10.1016/J.JCLINEPI.2004.03.017
An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods.
P. Devereaux (2004)
10.3310/HTA7270
Evaluating non-randomised intervention studies.
J. Deeks (2003)
10.1136/bmj.328.7430.22
Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group
H. Soares (2004)
10.1002/9781118394472.CH7
Bayesian Meta‐Analysis
Jegar Pitchforth (2012)
10.1002/bjs.5378
Meta‐analysis of flavonoids for the treatment of haemorrhoids
P. Alonso-Coello (2006)
10.1001/JAMA.282.11.1054
The hazards of scoring the quality of clinical trials for meta-analysis.
P. Jüni (1999)
10.1001/jama.2010.310
Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis.
D. Bassler (2010)
10.20344/AMP.1001
[Pharmacological therapy of Cushing's disease].
I. Paiva (2005)



This paper is referenced by
10.4324/9780203132678
Routledge handbook of physical activity and mental health
P. Ekkekakis (2013)
10.1007/s00268-012-1494-3
Somatostatin Analogues for the Treatment of Enterocutaneous Fistulas: A Systematic Review and Meta-analysis
S. Coughlin (2012)
10.1002/14651858.CD012181.pub2
Antibiotics at the time of removal of central venous catheter to reduce morbidity and mortality in newborn infants.
Rowena L McMullan (2018)
The cultural, organisational and contextual processes that might affect the implementation of massage in Lebanese neonatal intensive care units : a study informed by normalization process theory
B. Abdallah (2018)
10.1016/j.jclinepi.2014.11.023
Grading the strength of a body of evidence when assessing health care interventions: an EPC update.
N. Berkman (2015)
10.1002/14651858.CD011746.pub2
Gemcitabine-based chemotherapy for advanced biliary tract carcinomas.
O. Abdel-Rahman (2018)
10.1111/aos.12665
Cataract surgery and age‐related macular degeneration. An evidence‐based update
L. Kessel (2015)
10.1016/j.jclinepi.2015.03.008
Basic study design influences the results of orthodontic clinical investigations.
Spyridon N. Papageorgiou (2015)
10.1002/14651858.CD012039
Pharmacotherapy for hypertension‐induced left ventricular hypertrophy
R. M. Finizola (2016)
10.1111/all.13241
Vitamin D supplementation in primary allergy prevention: Systematic review of randomized and non‐randomized studies
J. Yepes-Nuñez (2018)
10.1007/978-3-319-23329-1_4
Evaluating Published CER Evidence
J. Ren (2016)
10.1186/s13643-019-0953-4
Comparative effectiveness of the different components of care provided in heart failure clinics—protocol for a systematic review and network meta-analysis
Morgan Slater (2019)
10.1186/s13643-018-0904-5
Screening for chlamydia and/or gonorrhea in primary health care: protocol for systematic review
Jennifer Pillay (2018)
10.1016/j.resuscitation.2017.10.021
2017 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science With Treatment Recommendations Summary.
T. Olasveengen (2017)
10.1503/cjs.001411
Management of chest tubes after pulmonary resection: a systematic review and meta-analysis.
S. M. Coughlin (2012)
10.1111/j.1467-789X.2012.01029.x
Nurse delivered lifestyle interventions in primary health care to treat chronic disease risk factors associated with obesity: a systematic review
G. Sargent (2012)
Health effects of water fluoridation
M. Sutton (2015)
10.1186/s13104-015-1181-1
Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study
S. O'Connor (2015)
10.1002/14651858.MR000050
Placebo, usual care and wait-list interventions for all mental health disorders
E. Faltinsen (2019)
10.1101/2020.04.17.20064469
Effectiveness and Safety of Glucocorticoids to Treat COVID-19: A Rapid Review and Meta-Analysis
S. Lu (2020)
10.1007/s11695-019-03732-8
Intravenous Acetaminophen Versus Placebo in Post-bariatric Surgery Multimodal Pain Management: a Meta-analysis of Randomized Controlled Trials
Y. Lee (2019)
10.1080/19411243.2018.1496868
Effectiveness of a 5-day summer handwriting camp for children ages 5 to 9 years old
Mackenzie Benning (2018)
10.3390/nu11020338
DASH Dietary Pattern and Cardiometabolic Outcomes: An Umbrella Review of Systematic Reviews and Meta-Analyses
L. Chiavaroli (2019)
10.1155/2013/583627
Serosorting and HIV/STI Infection among HIV-Negative MSM and Transgender People: A Systematic Review and Meta-Analysis to Inform WHO Guidelines
C. Kennedy (2013)
10.1080/10903127.2017.1384874
Effect of Task Load Interventions on Fatigue in Emergency Medical Services Personnel and Other Shift Workers: A Systematic Review
J. Studnek (2018)
10.1186/s13643-020-01360-3
The impact of vitamin D food fortification and health outcomes in children: a systematic review and meta-regression
Reem A Al Khalifah (2020)
10.1093/humupd/dmaa007
Reply: Sperm retrieval rates by micro-TESE versus conventional TESE in men with non-obstructive azoospermia-the assumption of independence in effects sizes might lead to misleading conclusions.
G. Corona (2020)
Systematic Review and Meta-Analysis of Pigtail Catheter Use in Traumatic Pneumothorax and Hemothorax
Anthony J. Carden (2017)
10.1093/jamia/ocz209
Understanding the nature and scope of clinical research commentaries in PubMed
James R. Rogers (2020)
10.30773/pi.2017.09.06
A Systematic Review of Nonpharmacological Interventions for Moderate to Severe Dementia: A Study Protocol for a Systematic Review and Meta-Analysis
Riyoung Na (2018)
10.1002/14651858.CD011724.pub2
Needle aspiration versus intercostal tube drainage for pneumothorax in the newborn.
M. Bruschettini (2016)
10.1177/0884533612474041
Critical reading and critical thinking--study design and methodology: a personal approach on how to read the clinical literature.
T. Lipman (2013)
See more
Semantic Scholar Logo Some data provided by SemanticScholar