Online citations, reference lists, and bibliographies.
← Back to Search

Effect Of Rater Training On Reliability And Accuracy Of Mini-CEX Scores: A Randomized, Controlled Trial

D. Cook, D. Dupras, T. Beckman, Kris G. Thomas, V. Pankratz
Published 2008 · Medicine

Save to my Library
Download PDF
Analyze on Scholarcy Visualize in Litmaps
Reduce the time it takes to create your bibliography by a factor of 10 by using the world’s favourite reference manager
Time to take this seriously.
Get Citationsy
BackgroundMini-CEX scores assess resident competence. Rater training might improve mini-CEX score interrater reliability, but evidence is lacking.ObjectiveEvaluate a rater training workshop using interrater reliability and accuracy.DesignRandomized trial (immediate versus delayed workshop) and single-group pre/post study (randomized groups combined).SettingAcademic medical center.ParticipantsFifty-two internal medicine clinic preceptors (31 randomized and 21 additional workshop attendees).InterventionThe workshop included rater error training, performance dimension training, behavioral observation training, and frame of reference training using lecture, video, and facilitated discussion. Delayed group received no intervention until after posttest.MeasurementsMini-CEX ratings at baseline (just before workshop for workshop group), and four weeks later using videotaped resident–patient encounters; mini-CEX ratings of live resident–patient encounters one year preceding and one year following the workshop; rater confidence using mini-CEX.ResultsAmong 31 randomized participants, interrater reliabilities in the delayed group (baseline intraclass correlation coefficient [ICC] 0.43, follow-up 0.53) and workshop group (baseline 0.40, follow-up 0.43) were not significantly different (p = 0.19). Mean ratings were similar at baseline (delayed 4.9 [95% confidence interval 4.6–5.2], workshop 4.8 [4.5–5.1]) and follow-up (delayed 5.4 [5.0–5.7], workshop 5.3 [5.0–5.6]; p = 0.88 for interaction). For the entire cohort, rater confidence (1 = not confident, 6 = very confident) improved from mean (SD) 3.8 (1.4) to 4.4 (1.0), p = 0.018. Interrater reliability for ratings of live encounters (entire cohort) was higher after the workshop (ICC 0.34) than before (ICC 0.18) but the standard error of measurement was similar for both periods.ConclusionsRater training did not improve interrater reliability or accuracy of mini-CEX scores.Clinical trials identifier NCT00667940
This paper references
Statistical Power Analysis for the Behavioral Sciences
Jacob Cohen (1969)
The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability
J. Fleiss (1973)
The measurement of observer agreement for categorical data.
J. R. Landis (1977)
The selection and training of examiners for clinical examinations
D. Newble (1980)
Clinical competence certification: a critical appraisal.
J. Woolliscroft (1984)
A Closer Look at Halo Error in Performance Ratings
R. Jacobs (1985)
A comparative trial of the clinical evaluation exercise.
F. Kroboth (1985)
Longitudinal data analysis for discrete and continuous outcomes.
S. Zeger (1986)
Statistical Inference
G. Casella (1990)
Standard Error of Measurement
L. Harvill (1991)
How well do internal medicine faculty members evaluate the clinical skills of residents?
G. Noel (1992)
Rater training for performance appraisal: A quantitative review
D. J. Woehr (1994)
The Mini-CEX (Clinical Evaluation Exercise): A Preliminary Investigation
J. Norcini (1995)
Evaluation of standardized rater training for the Positive and Negative Syndrome Scale (PANSS)
M. Müller (1998)
Sample size and optimal designs for reliability studies.
S. Walter (1998)
Methods for Evaluating the Clinical Competence of Residents in Internal Medicine: A Review
E. Holmboe (1998)
An Essay on the History and Future of Reliability from the Perspective of Replications
Robert L. Brennan (2001)
Standardized rater training for the Hamilton Depression Rating Scale (HAMD-17) in psychiatric novices.
M. Müller (2003)
Construct Validity of the MiniClinical Evaluation Exercise (MiniCEX)
E. Holmboe (2003)
The Mini-CEX: A Method for Assessing Clinical Skills
J. Norcini (2003)
SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings
R. Williams (2003)
Feasibility, Reliability, and Validity of the Mini-Clinical Evaluation Exercise (mCEX) in a Medicine Core Clerkship
J. Kogan (2003)
Effects of Training in Direct Observation of Medical Residents' Clinical Competence
E. Holmboe (2004)
Raters who pursue different goals give different ratings.
K. Murphy (2004)
Research in clinical reasoning: past history and current trends
G. Norman (2005)
Reporting of Ethical Committee Approval and Patient Consent by Study Design in 5 General Medical Journals
S Schroter (2005)
Assessing the mini‐Clinical Evaluation Exercise in comparison to a national specialty examination
R. Hatala (2006)
Effects of training intensity on observers' ratings of anxiety, social skills, and alcohol-specific coping skills.
A. Angkaw (2006)
Use of the Mini-Clinical Evaluation Exercise to Rate Examinee Performance on a Multiple-Station Clinical Skills Examination: A Validity Study
M. Margolis (2006)
Reporting ethics committee approval and patient consent by study design in five general medical journals
S. Schroter (2006)
Enriched rater training using Internet based technologies: a comparison to traditional rater training in a multi-site depression trial.
K. Kobak (2006)
Does feedback matter? Practice‐based learning for medical students after a multi‐institutional clinical performance examination
M. Srinivasan (2007)
Effectiveness of a focused educational intervention on resident evaluations from faculty
E. Holmboe (2001)
How accurate are faculty evaluations of clinical competence?
J. Herbers (1989)
Didactic value of the clinical evaluation exercise missed opportunities
F. Kroboth (1996)
The inter-rater reliability and internal consistency of a clinical evaluation exercise
Frank J. Kroboth (1992)
Identifying the factors that determine feedback given to undergraduate medical students following formative mini‐CEX assessments
N. Fernando (2007)
Psychometric properties of mini-clinical evaluation exercise (mini-CEX) scores: Accuracy, reliability, and effect of scale length
DA Cook (2008)

This paper is referenced by
Examen de l'exactitude des autoévaluations des résidents et des comportements d'évaluation des professeurs en anesthésiologie
Melinda Fleming (2021)
How Teachers Adapt Their Cognitive Strategies When Using Entrustment Scales.
Teaching residents how to break bad news: piloting a resident-led curriculum and feedback task force as a proof-of-concept study
Joseph Sleiman (2021)
The influence of candidates’ physical attributes on assessors’ ratings in clinical practice
A. Sam (2021)
Measuring the Effect of Examiner Variability in a Multiple-Circuit Objective Structured Clinical Examination (OSCE)
P. Yeates (2021)
Examining the accuracy of residents’ self-assessments and faculty assessment behaviours in anesthesiology
Melinda Fleming (2021)
Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation.
Examiners' Perceptions in Surgical Education: The Blind Spot in the Assessment of OSCEs.
Anna C. van der Want (2020)
Work-Based Assessments in Higher General Surgical Training Program: A Mixed Methods Study Exploring Trainers' and Trainees' Views and Experiences
K. Aryal (2020)
Evaluation of a new e-learning resource for calibrating OSCE examiners on the use of rating scales.
Rosa Moreno-López (2020)
Impact of Structured Feedback on Examiner Judgements in Objective Structured Clinical Examinations (OSCEs) Using Generalisability Theory
W. Y. Wong (2020)
Validity, reliability and feasibility of a new observation rating tool and a post encounter rating tool for the assessment of clinical reasoning skills of medical students during their internal medicine clerkship: a pilot study
C. M. Haring (2020)
What works best in a general practice specific OSCE for medical students: Mini-CEX or content-related checklists?
Patrick Giemsa (2020)
Assessing physical examination skills using direct observation and volunteer patients
B. Clark (2020)
Clinical assessors’ working conceptualisations of undergraduate consultation skills: a framework analysis of how assessors make expert judgements in practice
C. Hyde (2020)
Inter-rater reliability in clinical assessments: do examiner pairings influence candidate ratings?
Aileen Faherty (2020)
Teaching Geriatrics and Transitions of Care to Internal Medicine Resident Physicians
Shirley Wu (2020)
Rater Training in Medical Education: A Scoping Review
A. Vergis (2020)
Use of Mini Clinical Evaluation Exercise as a Tool to Assess the Orthodontic Postgraduate Students
S. Jamenis (2020)
National inter-rater agreement of standardised simulated-patient-based assessments
A. Sam (2020)
A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement
John Q Young (2020)
An investigation into the accessibility of Massive Open Online Courses (MOOCs)
Francisco Iniesto (2020)
The effectiveness of mini-CEX assessment tool for clinical competency achievement in clinical practice among anesthesia trainee.
Aamir Furqan (2020)
Faculty and Resident Perceptions on Mini-Clinical Examination Exercise (Mini-CEX) as an Assessment Tool in Medical and Surgical Super-Specialty Departments of a Teaching Hospital
S. Rajan (2020)
Review and Application of the Mini-Clinical Evaluation Exercise (Mini-CEX) in Advanced Orthodontic Education: A Pilot Study.
T. Al-Jewair (2019)
Comparison of two data capture methods and gender during clinical assessment in osteopathy: The impact on student/ tutor satisfaction ratings
P. Bright (2019)
Realizing One's Own Subjectivity: Assessors' Perceptions of the Influence of Training on Their Conduct of Workplace-Based Assessments.
K. Hodwitz (2019)
The Power of Subjectivity in the Assessment of Medical Trainees.
O. ten Cate (2019)
Milestone Ratings and Supervisory Role Categorizations Swim Together, but Is the Water Muddy?
D. Schumacher (2019)
Reliability of Observational Assessment Methods for Outcome-based Assessment of Surgical Skill: Systematic Review and Meta-analyses.
M. Groenier (2019)
A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments
Matthew Kelleher (2019)
Optimizing assessors’ mental workload in rater-based assessment: a critical narrative review
Bridget Paravattil (2019)
See more
Semantic Scholar Logo Some data provided by SemanticScholar