← Back to Search
Using Multiple Types Of Studies In Systematic Reviews Of Health Care Interventions – A Systematic Review
F. Peinemann, D. Tushabe, J. Kleijnen
Published 2013 · Medicine
Save to my Library
Download via 🐼 PaperPanda Download via oaDOI Download via OAB Download via LibKey Download via Google Google ScholarAnalyze on Scholarcy Visualize in Litmaps
Reduce the time it takes to create your bibliography by a factor of 10 by using the world’s favourite reference manager
Time to take this seriously.
Background A systematic review may evaluate different aspects of a health care intervention. To accommodate the evaluation of various research questions, the inclusion of more than one study design may be necessary. One aim of this study is to find and describe articles on methodological issues concerning the incorporation of multiple types of study designs in systematic reviews on health care interventions. Another aim is to evaluate methods studies that have assessed whether reported effects differ by study types. Methods and Findings We searched PubMed, the Cochrane Database of Systematic Reviews, and the Cochrane Methodology Register on 31 March 2012 and identified 42 articles that reported on the integration of single or multiple study designs in systematic reviews. We summarized the contents of the articles qualitatively and assessed theoretical and empirical evidence. We found that many examples of reviews incorporating multiple types of studies exist and that every study design can serve a specific purpose. The clinical questions of a systematic review determine the types of design that are necessary or sufficient to provide the best possible answers. In a second independent search, we identified 49 studies, 31 systematic reviews and 18 trials that compared the effect sizes between randomized and nonrandomized controlled trials, which were statistically different in 35%, and not different in 53%. Twelve percent of studies reported both, different and non-different effect sizes. Conclusions Different study designs addressing the same question yielded varying results, with differences in about half of all examples. The risk of presenting uncertain results without knowing for sure the direction and magnitude of the effect holds true for both nonrandomized and randomized controlled trials. The integration of multiple study designs in systematic reviews is required if patients should be informed on the many facets of patient relevant issues of health care interventions.
This paper references
comparing results of randomized trials and nonrandomized studies of interventions for low back pain
AD Furlan (1976)
Coronary artery surgery study (CASS): a randomized trial of coronary artery bypass surgery. Comparability of entry characteristics and survival in randomized patients and nonrandomized patients meeting randomization criteria.
Cass Principal Investigators and Their Associates (1984)
Asymptomatic cervical bruit and abnormal ocular pneumoplethysmography: a prospective study comparing two approaches to management.
G. Clagett (1984)
Selection bias in clinical trials.
K. Antman (1985)
Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery.
M. Hlatky (1988)
How study design affects outcomes in comparisons of therapy. I: Medical.
G. Colditz (1989)
D. Sackett (1994)
Worldwide Collaborative Observational Study and Meta‐Analysis on Allogenic Leukocyte Immunotherapy for Recurrent Spontaneous Abortion 1
C. Coulam (1994)
Worldwide collaborative observational study and meta-analysis on allogenic leukocyte immunotherapy for recurrent spontaneous abortion. Recurrent Miscarriage Immunotherapy Trialists Group.
When a randomised controlled trial is needed to assess drug safety. The case of paediatric ibuprofen
AA Mitchell (1995)
Why we need observational studies to evaluate the effectiveness of health care
N. Black (1996)
Randomization is important in studies with pain outcomes: systematic review of transcutaneous electrical nerve stimulation in acute postoperative pain.
D. Carroll (1996)
Primary total hip replacement surgery: a systematic review of outcomes and modelling of cost-effectiveness associated with different prostheses.
R. Fitzpatrick (1998)
The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials
R. Kunz (1998)
Observational research and evidence-based medicine: What should we teach young physicians?
J. Vandenbroucke (1998)
Choosing between randomised and non-randomised studies: a systematic review
M. Campbell (1998)
Can it work? Does it work? Is it worth it?
B. Haynes (1999)
Use of observational databases to evaluate the effectiveness of antiretroviral therapy for HIV infection: comparison of cohort studies with randomized trials. EuroSIDA, the French Hospital Database on HIV and the Swiss HIV Cohort Study Groups.
A. Phillips (1999)
Randomized trials versus observational studies in adolescent pregnancy prevention.
G. Guyatt (2000)
A comparison of observational studies and randomized, controlled trials.
K. Benson (2000)
Using routine data to complement and enhance the results of randomised controlled trials.
J. Lewsey (2000)
Effectiveness of exposure and ritual prevention for obsessive-compulsive disorder: randomized compared with nonrandomized samples.
M. Franklin (2000)
I. Witten (2000)
A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies.
R. MacLehose (2000)
Randomized, controlled trials, observational studies, and the hierarchy of research designs.
J. Concato (2000)
Bridging case-control studies and randomized trials
F. Rosendaal (2001)
Comparison of evidence of treatment effects in randomized and nonrandomized studies.
J. Ioannidis (2001)
Generalizability of trial results based on randomized versus nonrandomized allocation of OME infants to ventilation tubes or watchful waiting.
M. Rovers (2001)
Comparison of evidence of treatment effects in randomized and Multiple Study Designs PLOS ONE | www.plosone.org 11 December 2013 | Volume 8 | Issue 12 | e85035 nonrandomized studies
JP Ioannidis (2001)
Should systematic reviews include non-randomized and uncontrolled studies? The case of acupuncture for chronic headache.
K. Linde (2002)
Evaluation of interventions.
R. Fletcher (2002)
Are pacifiers associated with early weaning from breastfeeding?
M. Benis (2002)
Evaluating non-randomised intervention studies.
J. Deeks (2003)
Randomized controlled trials in evidence-based mental health care: getting the right answer to the right question.
S. Essock (2003)
Why results of clinical trials and observational studies of antitumour necrosis factor (anti-TNF) therapy differ: methodological and interpretive issues
F. Wolfe (2004)
The effectiveness of pneumococcal polysaccharide vaccines in adults: a systematic review of observational studies and comparison with results from randomised controlled trials.
S. Conaty (2004)
Evidence-based public health: moving beyond randomized trials.
C. Victora (2004)
Rare diseases and the assessment of intervention: What sorts of clinical trials can we use?
B. Wilcken (2004)
Hierarchy of evidence: differences in results between non-randomized studies and randomized trials in patients with femoral neck fractures
M. Bhandari (2004)
Those confounded vitamins: what can we learn from the differences between observational versus randomised trial evidence?
D. Lawlor (2004)
When are observational studies as credible as randomised trials?
J. Vandenbroucke (2004)
Common Pitfalls in the Conduct of Clinical Research
M. Zlowodzki (2005)
Contradicted and initially stronger effects in highly cited clinical research.
J. Ioannidis (2005)
Systematic reviews incorporating evidence from nonrandomized study designs: reasons for caution when estimating health effects
B. Reeves (2005)
Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go?
D. Ogilvie (2005)
Challenges in Systematic Reviews of Therapeutic Devices and Procedures
L. Hartling (2005)
Challenges in Using Nonrandomized Studies in Systematic Reviews of Treatment Interventions
S. Norris (2005)
The generalizability of observational data to elderly patients was dependent on the research question in a systematic review.
C. Gross (2005)
Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies
Panagiotis N. Papanikolaou (2006)
Identifying observational studies of surgical interventions in MEDLINE and EMBASE
C. Fraser (2006)
Comparison of Effects in Randomized Controlled Trials With Observational Studies in Digestive Surgery
Satoru Shikata (2006)
Levels of Evidence
C. Heneghan (2006)
The nature of the scientific evidence leading to drug withdrawals for pharmacovigilance reasons in France
P. Olivier (2006)
Randomisation to protect against selection bias in healthcare trials.
R. Kunz (2007)
Creating and Synthesizing Evidence With Decision Makers in Mind: Integrating Evidence From Clinical Trials and Other Study Designs
D. Atkins (2007)
Measurement of long-term outcomes in observational and randomised controlled trials
R. Hodgson (2007)
Study designs for effectiveness and translation research :identifying trade-offs.
S. Mercer (2007)
Should meta-analyses of interventions include observational studies in addition to randomized controlled trials? A critical examination of underlying principles.
I. Shrier (2007)
Effect of aspirin on long-term risk of colorectal cancer: consistent evidence from randomised and observational studies
E. Flossmann (2007)
British Doctors Aspirin T, the UKTIAAT
E Flossmann (2007)
Evidence from nonrandomized studies: a case study on the estimation of causal effects.
C. Schmoor (2008)
Examining Heterogeneity in Meta-Analysis: Comparing Results of Randomized Trials and Nonrandomized Studies of Interventions for Low Back Pain
A. Furlan (2008)
Observational Research, Randomised Trials, and Two Views of Medical Science
J. Vandenbroucke (2008)
Methodological quality and homogeneity influenced agreement between randomized trials and nonrandomized studies of the same intervention for back pain.
A. Furlan (2008)
Where Do We Find the Best Evidence?
C. McCarthy (2008)
Randomized clinical trials and observational studies: guidelines for assessing respective strengths and limitations.
E. Hannan (2008)
Chapter 13: Including non-randomized studies
B. Reeves (2008)
Assessing Risk of Bias in Included Studies
J. Higgins (2008)
Outcomes of patients who participate in randomized controlled trials compared to similar patients receiving similar interventions who do not participate.
G. Vist (2008)
Chapter 8: Assessing risk of bias in included studies
J. Higgins (2008)
Overall survival in the intervention arm of a randomized controlled screening trial for prostate cancer compared with a clinically diagnosed cohort.
A. Vis (2008)
Including Non‐Randomized Studies
B. Reeves (2008)
Recommendations for research design of telehealth studies.
N. Chumbler (2008)
Evidence from nonrandomized studies: a case study on the estimation of causal Multiple Study Designs PLOS ONE | www.plosone.org effects
C Schmoor (2008)
Evidence from nonrandomized studies: a case study on the estimation of causal Multiple Study Designs PLOS ONE | www.plosone.org 12 December 2013 | Volume 8 | Issue 12 | e85035 effects
C Schmoor (2008)
From randomized controlled trials to observational studies.
S. Silverman (2009)
Hierarchy of evidence: where observational studies fit in and why we need them.
Daniel J. Hoppe (2009)
Are there roles for observational database studies and structured quantification of expert opinion to answer therapy controversies in transplants?
R. Gale (2009)
Applicability and generalisability of published results of randomised controlled trials and non-randomised studies evaluating four orthopaedic procedures: methodological systematic review
L. Pibouleau (2009)
Can Observational Studies Provide a Realistic Alternative to Randomized Controlled Trials in Palliative Care?
G. Hadley (2009)
The HRT controversy: observational studies and RCTs fall in line
J. Vandenbroucke (2009)
Bisphosphonates and osteoporotic fractures: a cross-design synthesis of results among compliant/persistent postmenopausal women in clinical practice versus randomized controlled trials
M. Wilkes (2009)
Not only randomized controlled trials, but also case series should be considered in systematic reviews of rapidly developing technologies.
D. Chambers (2009)
Controversy over "contradiction": Should randomized trials always trump observational studies?
A. Fletcher (2009)
Oxford: Centre of Evidence-Based Medicine (CEBM)
Levels of evidence. Oxford: Centre of Evidence-Based Medicine (CEBM)
Practice-based evidence for clinical practice improvement: an alternative study design for evidence-based medicine.
S. Horn (2010)
Use of non-randomised evidence alongside randomised trials in a systematic review of endovascular aneurysm repair: strengths and limitations.
D. Chambers (2010)
Cochrane Handbook for Systematic Reviews of Interventions
Michele Tarsilla (2010)
Meta-analysis of well-designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials.
Ned S. Abraham (2010)
The effect of randomization to weight loss on total mortality in older overweight and obese adults: the ADAPT Study.
M. Shea (2010)
Reported effects in randomized controlled trials were compared with those of nonrandomized trials in cholecystectomy.
Dirk Müeller (2010)
Comparative Effectiveness Research: What Kind of Studies Do We Need?
J. Concato (2010)
Comparative effectiveness of prostate cancer treatments: evaluating statistical adjustments for confounding in observational data.
J. Hadley (2010)
Observational methods in comparative effectiveness research.
J. Concato (2010)
Does mesh offer an advantage over tissue in the open repair of umbilical hernias? A systematic review and meta-analysis
N. Aslani (2010)
AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program.
R. Chou (2010)
Use of nonrandomised evidence alongside randomised trials in a systematic review of endovascular aneurysm repair: strengths and limitations
D Chambers (2010)
Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study
I. Tzoulaki (2011)
Meta-analyses of Adverse Effects Data Derived from Randomised Controlled Trials as Compared to Observational Studies: Methodological Overview
S. Golder (2011)
Surfactant Lavage Therapy for Meconium Aspiration Syndrome: A Systematic Review and Meta-Analysis
H. Choi (2011)
Treatments effects from randomized trials and propensity score analyses were similar in similar populations in an example from cardiac surgery.
O. Kuss (2011)
Implications of the principle of question propagation for comparative-effectiveness and "data mining" research.
Mia Djulbegovic (2011)
Why do the results of randomised and observational studies differ?
J. Vandenbroucke (2011)
From efficacy to effectiveness in the face of uncertainty: indication creep and prevention creep.
B. Djulbegovic (2011)
Antidepressant Response in Major Depressive Disorder: A Meta-Regression Comparison of Randomized Controlled Trials and Observational Studies
F. Naudet (2011)
MeSH: Randomized Controlled Trials as Topic MeSH: Epidemiologic Study Characteristics as Topic
Observational studies in systemic reviews of comparative effectiveness: AHRQ and the Effective Health Care Program
SL Norris (2011)
Explanatory models are needed to integrate RCT and observational data with the patient's unique biology
Vijay Sharma (2012)
Laparoscopic vs. open cholecystectomy for cirrhotic patients: a systematic review and meta-analysis.
Y. Cheng (2012)
Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials
J. Savović (2012)
Meta-analytic comparison of randomized and nonrandomized studies of breast cancer surgery.
J. Edwards (2012)
Investigating outcomes associated with medication use during pregnancy: a review of methodological challenges and observational study designs.
L. Grzeskowiak (2012)
Effects of regular aspirin on long-term cancer incidence and metastasis: a systematic comparison of evidence from observational studies versus randomised trials.
A. Algra (2012)
Observational studies versus randomized controlled trials: avenues to causal inference in nephrology.
C. Kovesdy (2012)
Workload and surgeon's specialty for outcome after colorectal cancer surgery.
D. Archampong (2012)
Metaanalytic comparison of randomized and nonrandomized studies of breast cancer surgery
JP Edwards (2012)
J R Soc Med
British Doctors Aspirin T, the UKTIAAT (2007) Effect of aspirin on long-term risk of colorectal cancer: consistent evidence from randomised and observational studies
This paper is referenced by
Gait-assisted exoskeletons for children with cerebral palsy or spinal muscular atrophy: A systematic review.
Carlos Cumplido (2021)
Improving antibiotic use through behaviour change: a systematic review of interventions evaluated in low- and middle-income countries.
Carla Cuevas (2021)
A Systematic Review of Randomized Controlled Trials of School Based Interventions on Sexual Risk Behaviors and Sexually Transmitted Infections Among Young Adolescents in Sub-Saharan Africa
Nosipho Shangase (2021)
Medical librarians' knowledge and practices in locating clinical trials for systematic reviews
J. Westrick (2021)
Errors in the implementation, analysis, and reporting of randomization within obesity and nutrition research: a guide to their avoidance
Colby J. Vorland (2021)
THE EFFECT OF SECOND-GENERATION ANTIDEPRESSANT TREATMENT ON THE EXECUTIVE FUNCTIONS OF PATIENTS WITH MAJOR DEPRESSIVE DISORDER: A META-ANALYSIS STUDY WITH STRUCTURAL EQUATION MODELS
E. Gudayol-Ferré (2021)
Single versus splinted short implants at sinus augmented sites: A systematic review of the literature.
Kelvin Ian Afrashtehfar (2020)
Ginseng and Cancer-Related Fatigue: A Systematic Review of Clinical Trials
Mehdi Sadeghian (2020)
Estudio de la influencia de la actividad física y la fatiga en personas con esclerosis múltiple
M. Pareja (2020)
Association between factor V Leiden mutation and recurrent pregnancy loss in the middle east countries: a Newcastle–Ottawa meta-analysis
Bahareh Hamedi (2020)
A Systematic Review and Comparison of Neurocognitive Features of Late-Life Attention-Deficit/Hyperactivity Disorder and Dementia With Lewy Bodies
J. Prentice (2020)
Subarachnoid haemorrhage with negative initial neurovascular imaging: a systematic review and meta-analysis
M. Mohan (2019)
Exercise for Nonagenarians: A Systematic Review
K. Miller (2019)
Nordic walking for women with breast cancer: A systematic review.
M. A. Sánchez-Lastra (2019)
Corticosteroids and Cognition: A Meta-Analysis
C. E. Prado (2019)
Aspectos básicos sobre la lectura de revisiones sistemáticas y la interpretación de meta-análisis:
José Ernesto Fernández-Chinguel (2019)
The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
C. Kennedy (2019)
Clinical effectiveness of enzyme replacement therapy with galsulfase in mucopolysaccharidosis type VI treatment: Systematic review
D. Gomes (2019)
Exercise Interventions for Improving Flexibility in People with Multiple Sclerosis: A Systematic Review and Meta-Analysis
Marta Torres-Pareja (2019)
More consideration was needed when conducting non-randomised studies of interventions.
P. Dhiman (2019)
Pilates for people with multiple sclerosis: A systematic review and meta-analysis.
M. A. Sanchez-Lastra (2019)
A Classic Herbal Formula Guizhi Fuling Wan for Menopausal Hot Flushes: From Experimental Findings to Clinical Applications
Mingdi Li (2019)
Does Integrative Medicine Reduce Prescribed Opioid Use for Chronic Pain? A Systematic Literature Review.
Samah Hassan (2019)
Clinical Outcomes of Stereotactic Radiosurgery for Cerebral Arteriovenous Malformations in Pediatric Patients: Systematic Review and Meta-Analysis.
A. Börcek (2019)
Effectiveness of non-pharmacological nursing interventions to improve the quality of life of patients with idiopathic pulmonary fibrosis: A systematic review.
Yasuko Igai (2018)
A Systematic Review of Attention Biases in Opioid, Cannabis, Stimulant Use Disorders
M. Zhang (2018)
[Systematic Reviews and Meta-analyses - Reading, Understanding, Interpreting].
S. Scheidt (2018)
Prevention of obesity and diabetes in pregnancy: is it an impossible dream?
O. Langer (2018)
The effect of universal maternal antenatal iron supplementation on neurodevelopment in offspring: a systematic review and meta-analysis
C. Jayasinghe (2018)
Robot-assisted upper extremity rehabilitation for cervical spinal cord injuries: a systematic scoping review
Hardeep Singh (2018)
A meta-analysis of the effects of antidepressants on cognitive functioning in depressed and non-depressed samples
C. E. Prado (2018)
Pilates for women with breast cancer: A systematic review and meta-analysis.
A. Pinto-Carral (2018)See more