Online citations, reference lists, and bibliographies.
← Back to Search

Effect Of Rater Training On Reliability And Accuracy Of Mini-CEX Scores: A Randomized, Controlled Trial

D. Cook, D. Dupras, T. Beckman, Kris G. Thomas, V. Pankratz
Published 2008 · Medicine

Save to my Library
Download PDF
Analyze on Scholarcy Visualize in Litmaps
Share
Reduce the time it takes to create your bibliography by a factor of 10 by using the world’s favourite reference manager
Time to take this seriously.
Get Citationsy
BackgroundMini-CEX scores assess resident competence. Rater training might improve mini-CEX score interrater reliability, but evidence is lacking.ObjectiveEvaluate a rater training workshop using interrater reliability and accuracy.DesignRandomized trial (immediate versus delayed workshop) and single-group pre/post study (randomized groups combined).SettingAcademic medical center.ParticipantsFifty-two internal medicine clinic preceptors (31 randomized and 21 additional workshop attendees).InterventionThe workshop included rater error training, performance dimension training, behavioral observation training, and frame of reference training using lecture, video, and facilitated discussion. Delayed group received no intervention until after posttest.MeasurementsMini-CEX ratings at baseline (just before workshop for workshop group), and four weeks later using videotaped resident–patient encounters; mini-CEX ratings of live resident–patient encounters one year preceding and one year following the workshop; rater confidence using mini-CEX.ResultsAmong 31 randomized participants, interrater reliabilities in the delayed group (baseline intraclass correlation coefficient [ICC] 0.43, follow-up 0.53) and workshop group (baseline 0.40, follow-up 0.43) were not significantly different (p = 0.19). Mean ratings were similar at baseline (delayed 4.9 [95% confidence interval 4.6–5.2], workshop 4.8 [4.5–5.1]) and follow-up (delayed 5.4 [5.0–5.7], workshop 5.3 [5.0–5.6]; p = 0.88 for interaction). For the entire cohort, rater confidence (1 = not confident, 6 = very confident) improved from mean (SD) 3.8 (1.4) to 4.4 (1.0), p = 0.018. Interrater reliability for ratings of live encounters (entire cohort) was higher after the workshop (ICC 0.34) than before (ICC 0.18) but the standard error of measurement was similar for both periods.ConclusionsRater training did not improve interrater reliability or accuracy of mini-CEX scores.Clinical trials registrationclinicaltrials.gov identifier NCT00667940
This paper references
10.2307/2529115
Statistical Power Analysis for the Behavioral Sciences
Jacob Cohen (1969)
10.1177/001316447303300309
The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability
J. Fleiss (1973)
10.2307/2529310
The measurement of observer agreement for categorical data.
J. R. Landis (1977)
10.1111/j.1365-2923.1980.tb02379.x
The selection and training of examiners for clinical examinations
D. Newble (1980)
10.1097/00001888-198410000-00004
Clinical competence certification: a critical appraisal.
J. Woolliscroft (1984)
10.5465/256068
A Closer Look at Halo Error in Performance Ratings
R. Jacobs (1985)
10.1001/ARCHINTE.1985.00360060189029
A comparative trial of the clinical evaluation exercise.
F. Kroboth (1985)
10.2307/2531248
Longitudinal data analysis for discrete and continuous outcomes.
S. Zeger (1986)
10.2307/j.ctv301h9r.8
Statistical Inference
G. Casella (1990)
10.1111/J.1745-3992.1991.TB00195.X
Standard Error of Measurement
L. Harvill (1991)
10.7326/0003-4819-117-9-757
How well do internal medicine faculty members evaluate the clinical skills of residents?
G. Noel (1992)
10.1111/J.2044-8325.1994.TB00562.X
Rater training for performance appraisal: A quantitative review
D. J. Woehr (1994)
10.7326/0003-4819-123-10-199511150-00008
The Mini-CEX (Clinical Evaluation Exercise): A Preliminary Investigation
J. Norcini (1995)
10.1016/S0920-9964(98)00051-6
Evaluation of standardized rater training for the Positive and Negative Syndrome Scale (PANSS)
M. Müller (1998)
10.1002/(SICI)1097-0258(19980115)17:1<101::AID-SIM727>3.0.CO;2-E
Sample size and optimal designs for reliability studies.
S. Walter (1998)
10.7326/0003-4819-129-1-199807010-00011
Methods for Evaluating the Clinical Competence of Residents in Internal Medicine: A Review
E. Holmboe (1998)
10.1111/J.1745-3984.2001.TB01129.X
An Essay on the History and Future of Reliability from the Perspective of Replications
Robert L. Brennan (2001)
10.1016/S0165-0327(02)00097-6
Standardized rater training for the Hamilton Depression Rating Scale (HAMD-17) in psychiatric novices.
M. Müller (2003)
10.1097/00001888-200308000-00018
Construct Validity of the MiniClinical Evaluation Exercise (MiniCEX)
E. Holmboe (2003)
10.7326/0003-4819-138-6-200303180-00012
The Mini-CEX: A Method for Assessing Clinical Skills
J. Norcini (2003)
10.1207/S15328015TLM1504_11
SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings
R. Williams (2003)
10.1097/00001888-200310001-00011
Feasibility, Reliability, and Validity of the Mini-Clinical Evaluation Exercise (mCEX) in a Medicine Core Clerkship
J. Kogan (2003)
10.7326/0003-4819-140-11-200406010-00008
Effects of Training in Direct Observation of Medical Residents' Clinical Competence
E. Holmboe (2004)
10.1037/0021-9010.89.1.158
Raters who pursue different goals give different ratings.
K. Murphy (2004)
10.1111/j.1365-2929.2005.02127.x
Research in clinical reasoning: past history and current trends
G. Norman (2005)
Reporting of Ethical Committee Approval and Patient Consent by Study Design in 5 General Medical Journals
S Schroter (2005)
10.1111/j.1365-2929.2006.02566.x
Assessing the mini‐Clinical Evaluation Exercise in comparison to a national specialty examination
R. Hatala (2006)
10.1016/J.BRAT.2005.04.002
Effects of training intensity on observers' ratings of anxiety, social skills, and alcohol-specific coping skills.
A. Angkaw (2006)
10.1097/01.ACM.0000236514.53194.f4
Use of the Mini-Clinical Evaluation Exercise to Rate Examinee Performance on a Multiple-Station Clinical Skills Examination: A Validity Study
M. Margolis (2006)
10.1136/jme.2005.015115
Reporting ethics committee approval and patient consent by study design in five general medical journals
S. Schroter (2006)
10.1016/J.JPSYCHIRES.2005.07.012
Enriched rater training using Internet based technologies: a comparison to traditional rater training in a multi-site depression trial.
K. Kobak (2006)
10.1111/j.1365-2923.2007.02818.x
Does feedback matter? Practice‐based learning for medical students after a multi‐institutional clinical performance examination
M. Srinivasan (2007)
10.1046/j.1525-1497.2001.016007427.x
Effectiveness of a focused educational intervention on resident evaluations from faculty
E. Holmboe (2001)
10.1007/BF02599524
How accurate are faculty evaluations of clinical competence?
J. Herbers (1989)
10.1007/BF02599606
Didactic value of the clinical evaluation exercise missed opportunities
F. Kroboth (1996)
10.1007/BF02598008
The inter-rater reliability and internal consistency of a clinical evaluation exercise
Frank J. Kroboth (1992)
10.1111/j.1365-2923.2007.02939.x
Identifying the factors that determine feedback given to undergraduate medical students following formative mini‐CEX assessments
N. Fernando (2007)
Psychometric properties of mini-clinical evaluation exercise (mini-CEX) scores: Accuracy, reliability, and effect of scale length
DA Cook (2008)



This paper is referenced by
Examen de l'exactitude des autoévaluations des résidents et des comportements d'évaluation des professeurs en anesthésiologie
Melinda Fleming (2021)
10.1097/ACM.0000000000004287
How Teachers Adapt Their Cognitive Strategies When Using Entrustment Scales.
(2021)
10.1136/bmjstel-2021-000897
Teaching residents how to break bad news: piloting a resident-led curriculum and feedback task force as a proof-of-concept study
Joseph Sleiman (2021)
10.1080/0142159X.2021.1877268
The influence of candidates’ physical attributes on assessors’ ratings in clinical practice
A. Sam (2021)
10.1097/ACM.0000000000004028
Measuring the Effect of Examiner Variability in a Multiple-Circuit Objective Structured Clinical Examination (OSCE)
P. Yeates (2021)
10.36834/cmej.70697
Examining the accuracy of residents’ self-assessments and faculty assessment behaviours in anesthesiology
Melinda Fleming (2021)
10.4300/JGME-D-20-01110.1
Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation.
(2021)
10.1016/j.jsurg.2020.07.024
Examiners' Perceptions in Surgical Education: The Blind Spot in the Assessment of OSCEs.
Anna C. van der Want (2020)
10.1055/s-0040-1708062
Work-Based Assessments in Higher General Surgical Training Program: A Mixed Methods Study Exploring Trainers' and Trainees' Views and Experiences
K. Aryal (2020)
10.1111/eje.12495
Evaluation of a new e-learning resource for calibrating OSCE examiners on the use of rating scales.
Rosa Moreno-López (2020)
10.1016/j.hpe.2020.02.005
Impact of Structured Feedback on Examiner Judgements in Objective Structured Clinical Examinations (OSCEs) Using Generalisability Theory
W. Y. Wong (2020)
10.1186/s12909-020-02110-8
Validity, reliability and feasibility of a new observation rating tool and a post encounter rating tool for the assessment of clinical reasoning skills of medical students during their internal medicine clerkship: a pilot study
C. M. Haring (2020)
10.1080/0142159X.2020.1721449
What works best in a general practice specific OSCE for medical students: Mini-CEX or content-related checklists?
Patrick Giemsa (2020)
10.1515/dx-2019-0089
Assessing physical examination skills using direct observation and volunteer patients
B. Clark (2020)
10.1007/s10459-020-09960-3
Clinical assessors’ working conceptualisations of undergraduate consultation skills: a framework analysis of how assessors make expert judgements in practice
C. Hyde (2020)
10.1186/s12909-020-02009-4
Inter-rater reliability in clinical assessments: do examiner pairings influence candidate ratings?
Aileen Faherty (2020)
10.3390/geriatrics5040072
Teaching Geriatrics and Transitions of Care to Internal Medicine Resident Physicians
Shirley Wu (2020)
10.7759/cureus.11363
Rater Training in Medical Education: A Scoping Review
A. Vergis (2020)
10.1177/0301574219888041
Use of Mini Clinical Evaluation Exercise as a Tool to Assess the Orthodontic Postgraduate Students
S. Jamenis (2020)
10.1080/0142159X.2020.1845909
National inter-rater agreement of standardised simulated-patient-based assessments
A. Sam (2020)
10.1007/s40037-020-00587-z
A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement
John Q Young (2020)
10.21954/OU.RO.0001117A
An investigation into the accessibility of Massive Open Online Courses (MOOCs)
Francisco Iniesto (2020)
10.29309/tpmj/2020.27.02.3452
The effectiveness of mini-CEX assessment tool for clinical competency achievement in clinical practice among anesthesia trainee.
Aamir Furqan (2020)
10.4038/SEAJME.V14I2.219
Faculty and Resident Perceptions on Mini-Clinical Examination Exercise (Mini-CEX) as an Assessment Tool in Medical and Surgical Super-Specialty Departments of a Teaching Hospital
S. Rajan (2020)
10.21815/JDE.019.131
Review and Application of the Mini-Clinical Evaluation Exercise (Mini-CEX) in Advanced Orthodontic Education: A Pilot Study.
T. Al-Jewair (2019)
10.22024/UNIKENT/03/AJPP.883
Comparison of two data capture methods and gender during clinical assessment in osteopathy: The impact on student/ tutor satisfaction ratings
P. Bright (2019)
10.1097/ACM.0000000000002943
Realizing One's Own Subjectivity: Assessors' Perceptions of the Influence of Training on Their Conduct of Workplace-Based Assessments.
K. Hodwitz (2019)
10.1097/ACM.0000000000002495
The Power of Subjectivity in the Assessment of Medical Trainees.
O. ten Cate (2019)
10.1016/j.acap.2018.06.002
Milestone Ratings and Supervisory Role Categorizations Swim Together, but Is the Water Muddy?
D. Schumacher (2019)
10.1016/j.jsurg.2019.07.007
Reliability of Observational Assessment Methods for Outcome-based Assessment of Surgical Skill: Systematic Review and Meta-analyses.
M. Groenier (2019)
10.1097/ACM.0000000000002997
A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments
Matthew Kelleher (2019)
10.1007/s40037-019-00535-6
Optimizing assessors’ mental workload in rater-based assessment: a critical narrative review
Bridget Paravattil (2019)
See more
Semantic Scholar Logo Some data provided by SemanticScholar