Online citations, reference lists, and bibliographies.

Tools For Direct Observation And Assessment Of Clinical Skills Of Medical Trainees: A Systematic Review.

Jennifer R. Kogan, Eric Holmboe, Karen E Hauer
Published 2009 · Medicine, Psychology
Cite This
Download PDF
Analyze on Scholarcy
Share
CONTEXT Direct observation of medical trainees with actual patients is important for performance-based clinical skills assessment. Multiple tools for direct observation are available, but their characteristics and outcomes have not been compared systematically. OBJECTIVES To identify observation tools used to assess medical trainees' clinical skills with actual patients and to summarize the evidence of their validity and outcomes. DATA SOURCES Electronic literature search of PubMed, ERIC, CINAHL, and Web of Science for English-language articles published between 1965 and March 2009 and review of references from article bibliographies. STUDY SELECTION Included studies described a tool designed for direct observation of medical trainees' clinical skills with actual patients by educational supervisors. Tools used only in simulated settings or assessing surgical/procedural skills were excluded. Of 10 672 citations, 199 articles were reviewed and 85 met inclusion criteria. DATA EXTRACTION Two authors independently abstracted studies using a modified Best Evidence Medical Education coding form to inform judgment of key psychometric characteristics. Differences were reconciled by consensus. RESULTS A total of 55 tools were identified. Twenty-one tools were studied with students and 32 with residents or fellows. Two were used across the educational continuum. Most (n = 32) were developed for formative assessment. Rater training was described for 26 tools. Only 11 tools had validity evidence based on internal structure and relationship to other variables. Trainee or observer attitudes about the tool were the most commonly measured outcomes. Self-assessed changes in trainee knowledge, skills, or attitudes (n = 9) or objectively measured change in knowledge or skills (n = 5) were infrequently reported. The strongest validity evidence has been established for the Mini Clinical Evaluation Exercise (Mini-CEX). CONCLUSION Although many tools are available for the direct observation of clinical skills, validity evidence and description of educational outcomes are scarce.
This paper references
Use of videotaping to evaluate pediatric resident performance of health supervision examinations of infants.
David P. McCormick (1993)
10.1097/00001888-199710000-00038
Use of encounter cards for evaluation of residents in obstetrics
B G Brennan (1997)
Assessing resident competency in an outpatient setting.
Andrea L Wendling (2004)
10.1111/j.1365-2923.1991.tb00083.x
The observed long-case in clinical assessment.
David I Newble (1991)
10.1016/0002-9610(94)90107-4
A pilot experience with competency-based clinical skills assessment in a surgical clerkship.
Gary L. Dunnington (1994)
10.1097/ACM.0b013e318065be1e
Medical Education Research and IRB Review: An Analysis and Comparison of the IRB Review Process at Six Institutions
Liselotte N. Dyrbye (2007)
10.1097/00001888-200410001-00022
Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains.
K. Anders Ericsson (2004)
10.1046/j.1365-2923.2000.00490.x
Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the leicester assessment package.
Robert Kee McKinley (2000)
10.1080/0142159032000151907
AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician
James M. Shumway (2003)
10.1016/j.jss.2006.11.002
Reliability and usefulness of clinical encounter cards for a third-year surgical clerkship.
Melanie L. Richards (2007)
10.1007/BF02598008
The inter-rater reliability and internal consistency of a clinical evaluation exercise
Frank J. Kroboth (2007)
10.1046/j.1365-2923.2000.00477.x
Student perceptions of a new integrated course in clinical methods for medical undergraduates.
Adrian M Hastings (2000)
10.1111/j.1365-2923.1989.tb01545.x
The use of videorecordings of medical postgraduates in improving clinical skills.
Mudasir Ahmad Mir (1989)
10.1007/s10459-008-9147-x
Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX
David A. Cook (2009)
10.2307/2529115
Statistical Power Analysis for the Behavioral Sciences
Jacob Cohen (1969)
10.1111/j.1365-2923.1994.tb02530.x
The direct clinical examination: an alternative method for the assessment of clinical psychiatry skills in undergraduate medical students.
J. Price (1994)
10.1111/j.1365-2923.1990.tb02508.x
Assessment of general practice consultations: content validity of a rating scale.
Richard B Hays (1990)
10.1097/00001888-200308000-00018
Construct Validity of the MiniClinical Evaluation Exercise (MiniCEX)
E. Holmboe (2003)
10.1016/S0734-3299(08)70652-X
Debriefing in the intensive care unit: A feedback tool to facilitate bedside teaching
Todd Dorman (2008)
10.1097/ACM.0b013e31819f8031
Commentary: On regulation and medical education: sociology, learning, and accountability.
Steven J Durning (2009)
10.1007/BF02599524
How accurate are faculty evaluations of clinical competence?
Jerome E. Jr. Herbers (2007)
10.1111/j.1745-3992.1995.tb00881.x
STANDARDS OF VALIDITY AND THE VALIDITY OF STANDARDS IN PERFORMANCE ASSESSMENT
S. Messick (2005)
10.1097/00001888-198401000-00004
Evaluating the medical history: observation versus write-up review.
James O. Woolliscroft (1984)
10.1097/00001888-200406000-00002
Assessing Competence in Communication and Interpersonal Skills: The Kalamazoo II Report
Francesca Duffy (2004)
10.1016/j.ophtha.2005.06.006
The ophthalmic clinical evaluation exercise: reliability determination.
Karl C. Golnik (2005)
10.1056/NEJMra054784
Assessment in medical education.
Malcolm Cox (1978)
10.1080/10401330701366622
Feasibility, Reliability and User Satisfaction With a PDA-Based Mini-CEX to Evaluate the Clinical Skills of Third-Year Medical Students
Dario M. Torre (2007)
10.1097/00001888-199910000-00059
In-training evaluation during an internal medicine clerkship.
Rose Hatala (1999)
10.1197/j.aem.2006.01.030
Reliability of a core competency checklist assessment in the emergency department: the Standardized Direct Observation Assessment Tool.
P. Shayne (2006)
10.7326/0003-4819-140-11-200406010-00008
Effects of Training in Direct Observation of Medical Residents' Clinical Competence
E. Holmboe (2004)
10.1056/nejmra054784
Assessment in medical education.
Ronald M Epstein (2007)
A program for enhancing medical interviewing using video-tape feedback in the family practice residency.
Cassata Dm (1977)
10.1111/j.1365-2923.1984.tb01476.x
Evaluating a direct observation exercise in a psychiatric clerkship.
Paul S. Links (1984)
10.1007/s11606-008-0578-0
The State of Evaluation in Internal Medicine Residency
Saima I. Chaudhry (2008)
10.1097/00001888-200209000-00020
Assessing the Reliability and Validity of the Mini—Clinical Evaluation Exercise for Internal Medicine Residency Training
Steven J Durning (2002)
10.1097/00001888-200211000-00021
Implementation of the mini-CEX to evaluate medical students' clinical skills.
Jennifer R. Kogan (2002)
10.1093/fampra/8.1.10
Resident psychosocial performance: a brief report.
Johanna Shapiro (1991)
Development and validation of the Davis Observation Code.
Edward J. Callahan (1991)
10.1007/BF00596229
The assessment of professional competence: Developments, research and practical implications
C.P.M. van der Vleuten (1996)
10.1007/BF02599606
Didactic value of the clinical evaluation exercise missed opportunities
Frank J. Kroboth (2007)
10.1016/j.ophtha.2004.04.014
The Ophthalmic Clinical Evaluation Exercise (OCEX).
Karl C. Golnik (2004)
10.3109/01421598909146278
A model for evaluating the history taking and physical examination skills of medical students.
Herbert L. Stone (1988)
10.1136/pgmj.2005.040477
Practical experience of using directly observed procedures, mini clinical evaluation examinations, and peer observation in pre-registration house officer (FY1) trainees
Anna E. Morris (2006)
10.1111/j.1525-1497.2006.00436.x
Brief report: Use of the mini-clinical evaluation exercise in internal medicine core clerkships
Jennifer R. Kogan (2006)
Best Evidence in Medical Education . Appendix IIIA Prototype BEME Coding Sheet
SB Hulley (2007)
10.1016/S0002-9610(05)81289-9
Structured single-observer methods of evaluation for the assessment of ward performance on the surgical clerkship.
Gordon Dunnington (1990)
10.1080/01421590701352261
Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training
Alberto Alves de Lima (2007)
10.1016/j.amjmed.2005.10.036
Current concepts in validity and reliability for psychometric instruments: theory and application.
David A. Cook (2006)
10.1023/A:1015750009235
Assessing the Comparability of Standardized Patient and Physician Evaluations of Clinical Skills
John R Boulet (2002)
10.1016/S0738-3991(01)00141-0
Assessment of medical students' communicative behaviour and attitudes: estimating the reliability of the use of the Amsterdam attitudes and communication scale through generalisability coefficients.
J.C.J.M. de Haes (2001)
10.1111/j.1365-2929.2006.02445.x
Strengths and weaknesses in the consultation skills of senior medical students: identification, enhancement and curricular change.
Adrian M Hastings (2006)
10.1111/j.1745-3992.1999.tb00010.x
Validating Measures of Performance
Michael T. Kane (2005)
10.1046/j.1365-2923.2002.01294.x
Assessing health professionals.
Jim Crossley (2002)
10.1097/00001888-199605000-00087
Physician‐patient communication in ambulatory settings
Craig S. Roth (1996)
10.1111/j.1365-2929.2004.01840.x
Validation of a checklist to assess ward round performance in internal medicine.
Kirsten Nørgaard (2004)
2008 AAMC Graduation Questionnaire Program Evaluation Survey : All Schools Summary Report Final
WP Burdick
Bedside encounter and clinical performance of junior clinical clerks.
Dawson Dj (1983)
10.1016/S0002-9610(02)00786-9
An encounter card system for increasing feedback to students.
J. Paukert (2002)
A psychometric study of measures of medical interviewing skills.
David B. Swanson (1981)
The Mini Clinical Evaluation Exercise
John J Norcini (2005)
10.1080/01421590500237895
A PDA-based instructional tool to monitor students’ cardiac auscultation during a medicine clerkship
Dario M. Torre (2005)
10.1111/j.1365-2929.2004.02019.x
Feasibility and reliability of an in-training assessment programme in an undergraduate clerkship.
Hester E. M. Daelmans (2004)
10.1197/aemj.10.3.219
Evaluation of emergency medicine resident death notification skills by direct observation.
Ronald S. Benenson (2003)
10.7326/0003-4819-138-6-200303180-00012
The Mini-CEX: A Method for Assessing Clinical Skills
J. Norcini (2003)
A program for enhancing medical interviewing using video-tape feedback in the family practice residency.
Donald M. Cassata (1977)
10.1111/j.1365-2923.1976.tb00436.x
Evaluation of the clinical performance and skill in paediatrics of medical students.
B. Lagerkvist (1976)
10.1111/j.1365-2923.1990.tb00013.x
A new method to evaluate clinical performance and critical incidents in anaesthesia: quantification of daily comments by teachers.
M. F. Rhoton (1990)
10.1111/j.1553-2712.1995.tb03108.x
Observation of emergency medicine residents at the bedside: how often does it happen?
William P. Burdick (1995)
10.1111/j.1525-1497.2004.30134.x
Feedback and the mini clinical evaluation exercise
Eric Holmboe (2007)
10.1108/et.1999.00441aab.021
Evaluation of Training.
L. Sjodahl (1975)
Structured clinical observations: a method to teach clinical skills with limited time and financial resources.
J Lindsey Lane (2000)
10.1111/j.1365-2923.2004.01747.x
Acquisition of communication skills in postgraduate training for general practice.
Anneke W M Kramer (2004)
10.5694/J.1326-5377.2008.TB01951.X
The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates.
B. Nair (2008)
10.7326/0003-4819-142-12_Part_2-200506211-00008
Challenges in Systematic Reviews of Educational Intervention Studies
D. Reed (2005)
10.1197/aemj.9.11.1338
Direct observation for assessing emergency medicine core competencies: interpersonal skills.
Nicholas Jouriles (2002)
10.1111/j.1365-2929.2006.02566.x
Assessing the mini-Clinical Evaluation Exercise in comparison to a national specialty examination.
Rose Hatala (2006)
10.1111/j.1365-2929.2006.02506.x
Assessing resident's knowledge and communication skills using four different evaluation tools.
Jim Nuovo (2006)
10.1197/aemj.9.11.1342
Protected clinical teaching time and a bedside clinical evaluation instrument in an emergency medicine training program.
Philip Shayne (2002)
10.1097/00001888-197605000-00009
Skills in history-taking and physical examination.
John F Aloia (1976)
10.1097/00001888-200005000-00046
Enhancing feedback to students using the mini-CEX (Clinical Evaluation Exercise).
Karen E Hauer (2000)
10.1080/01421590701291469
Developing scholarly projects in education: A primer for medical teachers
Thomas J. Beckman (2007)
10.1111/j.1365-2929.2004.01932.x
Reliability: on the reproducibility of assessment data.
Steven M. Downing (2004)
A qualitative study of the impact on learning of the Mini OBSERVATION AND ASSESSMENT OF MEDICAL TRAINEES ©2009 American Medical Association
A Alves de Lima (2005)
Designing Clinical Research
T. Newman (2006)
10.1016/j.cursur.2004.07.016
Assessing the competencies in general surgery residency training.
Cheryl I. Anderson (2005)
10.1097/00001888-198607000-00006
Evaluation of system providing feedback to students on videotaped patient encounters.
Peter C. Scheidt (1986)
Clinical skills. Quantitative measurement.
Stanley L. Wiener (1976)
10.1097/00001888-200410000-00010
A Call for Outcomes Research in Medical Education
Frederick M. Chen (2004)
10.1007/BF02602308
Measurement and improvement of humanistic skills in first-year trainees
Howard Beckman (2007)
10.1001/jama.294.9.1052
Costs and funding for published medical education research.
Darcy A. Reed (2005)
10.1097/00001888-200310001-00011
Feasibility, Reliability, and Validity of the Mini-Clinical Evaluation Exercise (mCEX) in a Medicine Core Clerkship
Jennifer R. Kogan (2003)
10.1097/00001888-198410000-00004
Clinical competence certification: a critical appraisal.
James O. Woolliscroft (1984)
Summative assessment: a pilot project in the west of Scotland.
L Malcolm Campbell (1993)
10.1111/j.1365-2929.2005.02094.x
Assessing professional competence: from methods to programmes
Cees van der Vleuten (2005)
10.1046/j.1365-2923.2003.01438.x
Reliability and validity of the direct observation clinical encounter examination (DOCEE).
Hossam El-Din Hamdy (2003)
10.1007/s11606-008-0842-3
Effect of Rater Training on Reliability and Accuracy of Mini-CEX Scores: A Randomized, Controlled Trial
David A. Cook (2008)
10.1111/J.1365-2923.2007.02939.X
Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments.
Nishan Fernando (2008)
10.1016/j.acra.2004.04.016
Patient, faculty, and self-assessment of radiology resident performance: a 360-degree method of measuring professionalism and interpersonal/communication skills.
Jonathan G. M. Wood (2004)
10.1001/archinte.1985.00360060189029
A comparative trial of the clinical evaluation exercise.
Frank J. Kroboth (1985)
10.1080/01421590400013529
A qualitative study of the impact on learning of the mini clinical evaluation exercise in postgraduate training
Alberto Alves de Lima (2005)
10.1016/0002-9610(91)90086-S
A model for the assessment of students' physician-patient interaction skills on the surgical clerkship.
Edward Hartman Reisner (1991)
10.1097/00001888-198905000-00018
Evaluating the interview performance of internal medicine interns
John R. Meuleman (1989)
10.1097/ACM.0b013e3181971fae
Commentary: the ACGME: public advocacy before resident advocacy.
Thomas J. Nasca (2009)
10.7196/SAMJ.1128
Formative assessment promotes learning in undergraduate clinical clerkships.
Vanessa Celeste Burch (2006)
10.1542/peds.2007-2637
Continuity-Structured Clinical Observations: Assessing the Multiple-Observer Evaluation in a Pediatric Resident Continuity Clinic
Karen P. Zimmer (2008)
10.1080/01421590801946962
Internal medicine residents’ perceptions of the Mini-Clinical Evaluation Exercise
Samir Malhotra (2008)
10.7326/0003-4819-123-10-199511150-00008
The Mini-CEX (Clinical Evaluation Exercise): A Preliminary Investigation
J. Norcini (1995)
The future of residency education: implementing a competency-based educational model.
Frederick D. Edwards (2007)
10.1016/S0736-4679(98)00181-4
Assessing emergency medicine resident communication skills using videotaped patient encounters: gaps in inter-rater reliability.
Steven T. Rosenzweig (1999)
10.1097/00001888-200203000-00021
A Clinical‐performance Biopsy Instrument
Robert Gaylon Ross (2002)
10.15766/mep_2374-8265.176
The Palliative Care Clinical Evaluation Exercise
Paul K. J. Han (2006)
Bedside encounter and clinical performance of junior clinical clerks.
David J. Dawson (1983)
10.1111/j.1365-2923.2008.03230.x
Evaluation of a novel assessment form for observing medical residents: a randomised, controlled trial.
Anthony A Donato (2008)
10.1111/j.1553-2712.1996.tb03448.x
Evaluation of resident performance and intensive bedside teaching during direct observation.
Rita Kay Cydulka (1996)
10.1046/j.1365-2923.2003.01594.x
Validity: on meaningful interpretation of assessment data.
S. Downing (2003)
10.1080/01421590500237895
A PDA-based instructional tool to monitor students’ cardiac auscultation during a medicine clerkship
Italo Masiello (2005)
10.7326/0003-4819-117-9-757
How well do internal medicine faculty members evaluate the clinical skills of residents?
G. Noel (1992)
Evaluation of Training
M. Reeves (1993)



This paper is referenced by
10.1097/ACM.0000000000002768
Management Reasoning: Implications for Health Professions Educators and a Research Agenda.
David A. Cook (2019)
10.1186/s12909-020-1932-0
Identity matters - perceptions of inter-professional feedback in the context of workplace-based assessment in Diabetology training: a qualitative study
Katrin Feller (2020)
10.1007/978-3-030-26849-7_23
Simulation in Trauma/Advanced Cardiac Life Support
Cesar R. Padilla (2020)
Using Modified Direct Observation of Procedural Skills (DOPS) to assess undergraduate medical students
Arezou Farajpour (2018)
Programmatic assessment of competency-based workplace learning
Harold G J Bok (2018)
10.1016/j.acap.2015.09.007
Targeting Simulation-Based Assessment for the Pediatric Milestones: A Survey of Simulation Experts and Program Directors.
Leah A. Mallory (2016)
Pilotstudie zum llen Feedback in der pädiatrischen Weiterbildung
Berthold Koletzko (2015)
Using 360-degree multi-source feedback to evaluate professionalism in surgery departments: An Iranian perspective
Adel Yazdankhah (2015)
10.1111/acem.12033
Assessing medical knowledge of emergency medicine residents.
Nikhil Goyal (2012)
10.3109/0142159X.2014.917160
Meeting international standards: A cultural approach in implementing the mini-CEX effectively in Indonesian clerkships
Yoyo Suhoyo (2014)
10.1002/9781118728130.CH36
Workplace‐Based Assessment
J. M. Kidd (2015)
10.1111/IGC.0b013e3181fc39d9
Teaching Cervical Cancer Surgery in Low- or Middle-Resource Countries
Laurie M. Elit (2010)
Confiabilidad y validez de un instrumento de evaluación de la competencia clínica integral de médicos residentes
Ana Carolina Sepúlveda Vildósola (2015)
10.1097/ACM.0000000000002294
Asking for Less and Getting More: The Impact of Broadening a Rater’s Focus in Formative Assessment
Walter Tavares (2018)
10.1080/08897077.2011.640172
A SBIRT Curriculum for Medical Residents: Development of a Performance Feedback Tool to Build Learner Confidence
Jennifer E. Hettema (2012)
Peer-assessment within dental education
Jorge Tricio-Pesce (2014)
10.1111/j.1365-2923.2011.04016.x
Respecting the expertise of clinician assessors: construct alignment is one good answer.
Shiphra Ginsburg (2011)
10.1016/j.jsurg.2011.07.015
Implementation of an objective structured clinical exam (OSCE) into orthopedic surgery residency training.
Michael J. Griesser (2012)
10.15766/mep_2374-8265.10648
Longitudinal Faculty Development Program to Promote Effective Observation and Feedback Skills in Direct Clinical Observation
Sheira L. Schlair (2017)
10.1071/HC11298
Clinical teachers working in primary care: what would they like changed in the medical school?
Susan J Hawken (2011)
10.1001/jama.2010.19
Tools to assess clinical skills of medical trainees.
Isabelle Pitrou (2010)
10.11401/JSVS.18-00106
血管外科医のための自作Off-the-Job Training—システムの分析と評価—
淳 仁田 (2018)
A Review on Comprehensive Development of Conceptual Framework for Scoring Tool of Professionalism Assessment Among Medical Undergraduates in Malaysia
AF Jusoh (2018)
10.5176/2345-718X_1.1.17
Development of Measurement Tool Mini-CEX (Mini Clinical Evaluation Exercise) as an Evaluation Tool of Nursing Students in Teaching Hospital of Universitas Muhammadiyah Yogyakarta
Syafrina Arbaani Djuria (2013)
10.1186/2054-314X-1-9
Naval casualty management training using human patient simulators
Itamar Netzer (2015)
Issues Related to the Objectivity of Student Assessment in Medical Education
Kyung-Seok Min (2013)
10.1007/s11606-012-2264-5
Patient Outcomes in Simulation-Based Medical Education: A Systematic Review
Benjamin Zendejas (2012)
10.1111/medu.12220
Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review.
J Eric Jelovsek (2013)
10.1212/WNL.0000000000002053
Neurology objective structured clinical examination reliability using generalizability theory
Angela D. Blood (2015)
10.1016/J.ARBR.2015.08.004
Provision of care by medical residents and the impact on quality.
Estefania Llopis Pastor (2015)
10.1016/J.PCL.2019.03.010
The Clinical Learning Environment and Workplace-Based Assessment: Frameworks, Strategies, and Implementation.
Duncan Henry (2019)
10.1177/0141076819848113
Assessing professional competence: a critical review of the Annual Review of Competence Progression
Katherine Woolf (2019)
See more
Semantic Scholar Logo Some data provided by SemanticScholar