Online citations, reference lists, and bibliographies.
Please confirm you are human
(Sign Up for free to never see this)
← Back to Search

Systematic Review Of The Empirical Evidence Of Study Publication Bias And Outcome Reporting Bias

K. Dwan, D. Altman, J. Arnaiz, J. Bloom, A. Chan, Eugenia Cronin, E. Decullier, P. Easterbrook, E. von Elm, C. Gamble, D. Ghersi, J. Ioannidis, J. Simes, P. Williamson
Published 2008 · Medicine

Save to my Library
Download PDF
Analyze on Scholarcy
Share
Background The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. Methodology/Principal Findings We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. Conclusions Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.
This paper references
10.1073/PNAS.98.3.831
Evolution of treatment effects over time: empirical insight from recursive cumulative metaanalyses.
J. Ioannidis (2001)
Systematic reviews in health care : meta-analysis in context
M. Egger (2001)
10.1136/bmj.280.6217.833
Study of information submitted by drug companies to licensing authorities.
E. Hemminki (1980)
10.1001/JAMA.287.21.2801
Quality of reporting of randomized trials as a measure of methodologic quality.
Karin Huwiler-Müntener (2002)
10.1001/JAMA.293.8.970
Evaluation of new treatments in radiation oncology: are they better than standard treatments?
H. Soares (2005)
10.1136/bmj.326.7400.1171
Evidence b(i)ased medicine—selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications
H. Melander (2003)
10.1002/SIM.2025
Identification and impact of outcome selection bias in meta-analysis.
P. Williamson (2005)
10.1186/1745-6215-8-9
Application and investigation of a bound for outcome reporting bias
P. Williamson (2007)
10.1016/S0140-6736(03)12799-7
Role of a research ethics committee in follow-up and publication of results
J. Pich (2003)
10.1016/J.JCLINEPI.2006.06.016
Reporting in randomized clinical trials improved after adoption of the CONSORT statement.
R. Kane (2007)
10.1111/1467-9876.00197
Bias in meta‐analysis due to outcome variable selection within studies
J. L. Hutton (2000)
10.1136/bmj.38488.385995.8F
Fate of biomedical research protocols and publication bias in France: retrospective cohort study
E. Decullier (2005)
10.1002/14651858.MR000005.PUB3
Full publication of results initially presented in abstracts.
R. Scherer (2007)
10.1186/1471-2458-6-165
Impact of funding on biomedical research: a retrospective cohort study
E. Decullier (2006)
10.1017/S0266462304001175
Factors influencing the publication of health research.
Eugenia Cronin (2004)
10.1503/cmaj.1041086
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
A. Chan (2004)
10.1097/01.FPC.0000236332.11304.8F
An empirical evaluation of multifarious outcomes in pharmacogenetics: beta-2 adrenoceptor gene polymorphisms in asthma treatment
D. Contopoulos-Ioannidis (2006)
10.1093/AJE/154.9.873
Effect of early patient enrollment on the time to completion and publication of randomized controlled trials.
A. Haidich (2001)
Issues in the design, conduct and reporting of clinical trials that impact on the quality of decision making
D. Ghersi (2006)
10.1002/14651858.MR000006.PUB2
Publication bias in clinical trials
S. Hopewell (2007)
10.1016/0140-6736(91)90201-Y
Publication bias in clinical research
P. Easterbrook (1991)
10.1002/14651858.MR000006
Publication bias in clinical trials (Protocol)
K. Loudon (2001)
10.1136/bmj.328.7430.22
Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group
H. Soares (2004)
10.1001/JAMA.287.21.2825
Publication bias in editorial decision making.
C. Olson (2002)
10.1055/S-2007-991648
[Evaluation of clinical trials following an approval from a research ethics committee].
S. Menzel (2007)
10.1002/0470870168
Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments
H. Rothstein (2006)
10.1136/bmj.38478.497164.F7
Systematic review of publication bias in studies on publication bias
H. Dubben (2005)
10.1136/bmj.38356.424606.8F
Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors
A. Chan (2005)
NIH clinical trials and publication bias.
K. Dickersin (1993)
10.1016/J.JCLINEPI.2004.02.018
Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time.
T. Trikalinos (2004)
10.1037/1082-989X.2.4.447
Finding the Missing Science: The Fate of Studies Submitted for Review by a Human Subjects Committee
H. Cooper (1997)
10.7892/BORIS.27773
Publication and non-publication of clinical trials: longitudinal study of applications submitted to a research ethics committee.
E. von Elm (2008)
10.1001/JAMA.291.20.2457
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
A. Chan (2004)
10.1002/9780470693926
Systematic Reviews in Health Care
M. Egger (2001)
10.1001/JAMA.280.3.250
Publication bias and research on passive smoking: comparison of published and unpublished studies.
A. L. Misakian (1998)
10.3310/HTA4100
Publication and related biases.
F. Song (2000)
How important is publication bias? A synthesis of available data.
K. Dickersin (1997)
10.1016/0197-2456(87)90155-3
Publication bias and clinical trials.
K. Dickersin (1987)
10.1093/JNCI/88.3-4.206
False-positive results in clinical trials: multiple significance tests and the problem of unreported comparisons.
I. Tannock (1996)
10.1191/0962280205sm415oa
Outcome selection bias in meta-analysis
P. Williamson (2005)
10.1136/bmj.315.7109.640
Publication bias: evidence of delayed publication in a cohort study of clinical research projects
J. M. Stern (1997)
10.1001/JAMA.1992.03480030052036
Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards.
K. Dickersin (1992)
10.1001/JAMA.279.4.281
Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials.
J. Ioannidis (1998)
10.1046/J.1365-2753.2002.00314.X
Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee.
S. Hahn (2002)
10.1046/J.1365-2125.1998.00759.X
Bias in reporting clinical trials.
A. Bardy (1998)



This paper is referenced by
Evidence-based medicine and clinical practice
M. Huic (2008)
10.1258/jtt.2012.111009
A systematic review of the mediating role of knowledge, self-efficacy and self-care behaviour in telehealth patients with heart failure
Y. Ciere (2012)
10.1179/2047480612Z.00000000085
Implications of clinical trial data sharing for medical writers
J. Ross (2013)
10.1002/jrsm.45
Individual patient data meta-analysis of time-to-event outcomes: one-stage versus two-stage approaches for estimating the hazard ratio under a random effects model.
J. Bowden (2011)
10.3928/02793695-20110705-06
What you see depends on where you're looking and how you look at it: publication bias and outcome reporting bias.
R. Howland (2011)
10.1002/14651858.CD008386.pub3
Statins for multiple sclerosis.
J. Wang (2011)
10.1016/J.RECESP.2010.09.002
Prevención secundaria: el reto permanente
V. Roger (2011)
Revisión de las tablas de composición de alimentos usadas para estimar la ingesta de nutrientes en Ecuador Revision of food composition tables used to estimate nutrient intake in Ecuador
Rocío Ortíz-Moncada (2016)
10.1080/00224499.2012.750639
Sexual Orientation and Gender Identity/Expression Related Peer Victimization in Adolescence: A Systematic Review of Associated Psychosocial and Health Outcomes
K. Collier (2013)
The Open University ’ s repository of research publications and other research outputs Agile challenges in practice : a thematic analysis
Peggy Gregory (2015)
T . Denagamage and A . M . O ' Connor C . K . Irwin , K . J . Yoon , C . Wang ,
Chong Qing Wang (2017)
Publications for Robert Simes
R. Simes (2016)
10.1101/104786
Statistical Correction of the Winner’s Curse Explains Replication Variability in Quantitative Trait Genome-Wide Association Studies
C. Palmer (2017)
10.1016/j.drugalcdep.2013.04.014
Mind-body practices: an alternative, drug-free treatment for smoking cessation? A systematic review of the literature.
Laura Carim-Todd (2013)
10.1016/J.RHUM.2011.11.006
Conclusions erronées des résumés dans les essais cliniques contrôlés en rhumatologie : comparaisons entre les conclusions et la partie résultats des résumés
S. Mathieu (2012)
10.1371/journal.pone.0137667
Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews
N. Pandis (2015)
10.1136/bmj.j3436
Tackling submission and publication bias
Rainer Spiegel (2017)
10.1111/jep.12147
How evidence-based medicine is failing due to biased trials and selective publication
Susanna Every-Palmer MBChB Franzcp MSc Consultant Forensic Psychiatrist (2014)
10.1002/phar.1467
Regulators Should Better Leverage Effectiveness Standards to Enhance Drug Value
H. Naci (2014)
10.1016/j.jclinepi.2018.05.023
Pre-specification of statistical analysis approaches in published clinical trial protocols was inadequate.
L. Greenberg (2018)
10.1073/pnas.1618569114
Meta-assessment of bias in science
D. Fanelli (2017)
10.1684/BDC.2012.1651
Enregistrement et publication du critère d’évaluation principal dans les essais contrôlés randomisés en oncologie
A. Boespflug (2012)
The impact of physical activity and blood pressure on cardiovascular events and mortality
A. Rossi (2015)
10.1201/B11349-14
Nonrandomized Studies to Evaluate the Effects of a Nonpharmacological Intervention
B. Reeves (2011)
10.1371/journal.pone.0006276
Outcomes in Clinical Trials of Inhaled Corticosteroids for Children with Asthma Are Narrowly Focussed on Short Term Disease Activity
I. Sinha (2009)
10.1080/17483107.2020.1801865
Towards improving the quality of assistive technology outcomes research.
Joshua R Tuazon (2020)
10.1016/B978-0-12-418689-7.00003-X
Chapter 3 – Study Design
Ü. Maiväli (2015)
10.1016/j.critrevonc.2017.02.008
A minimum core outcome dataset for the reporting of preclinical chemotherapeutic drug studies: Lessons learned from multiple discordant methodologies in the setting of colorectal cancer.
M. West (2017)
10.1145/3014812.3014872
Vulnerabilities associated with wi-fi protected setup in a medical environment
D. Costantin (2017)
10.1007/978-3-319-98902-0_13
Intersphincteric resection: indications and outcome.
F. McDermott (2019)
10.1016/j.jclinepi.2015.11.016
Reporting quality of N-of-1 trials published between 1985 and 2013: a systematic review.
J. Li (2016)
10.1016/j.gloepi.2020.100040
Epidemiology of PCBs and neurodevelopment: Systematic assessment of multiplicity and completeness of reporting
M. Goodman (2020)
See more
Semantic Scholar Logo Some data provided by SemanticScholar