Online citations, reference lists, and bibliographies.

The Inter-rater Reliability And Internal Consistency Of A Clinical Evaluation Exercise

Frank J. Kroboth, Barbara H. Hanusa, S. Parker, John L. Coulehan, Wishwa N. Kapoor, Frank H. Brown, M. Karpf, Gerald S. Levey
Published 2007 · Medicine

Cite This
Download PDF
Analyze on Scholarcy
Share
Objective:To assess the internal consistency and interrater reliability of a clinical evaluation exercise (CEX) format that was designed to be easily utilized, but sufficiently detailed, to achieve uniform recording of the observed examination.Design:A comparison of 128 CEXs conducted for 32 internal medicine interns by full-time faculty. This paper reports alpha coefficients as measures of internal consistency and several measures of inter-rater reliability.Setting:A university internal medicine program. Observations were conducted at the end of the internship year.Participants:Participants were 32 interns and observers were 12 full-time faculty in the department of medicine. The entire intern group was chosen in order to optimize the spectrum of abilities represented. Patients used for the study were recruited by the chief resident from the inpatient medical service based on their ability and willingness to participate.Intervention:Each intern was observed twice and there were two examiners during each CEX. The examiners were given a standardized preparation and used a format developed over five years of previous pilot studies.Measurements and main results:The format appeared to have excellent internal consistency; alpha coefficients ranged from 0.79 to 0.99. However, multiple methods of determining inter-rater reliability yielded similar results; intraclass correlations ranged from 0.23 to 0.50 and generalizability coefficients from a low of 0.00 for the overall rating of the CEX to a high of 0.61 for the physical examination section. Transforming scores to eliminate rater effects and dichotomizing results into pass-fail did not appear to enhance the reliability results.Conclusions:Although the CEX is a valuable didactic tool, its psychometric properties preclude reliable assessment of clinical skills as a one-time observation.
This paper references
10.1001/archinte.1990.00390150069014
Reliability and validity of an objective structured clinical examination for assessing the clinical performance of residents.
E. Petrusa (1990)
Theassessmentofclinicalskillsingeneral medical residents - - comparison of the objective structured clinical examination to a conventional oral examination
RothmanAI RobbKV (1985)
10.5694/j.1326-5377.1974.tb71187.x
General practitioners and psychosocial problems: An evaluation using pseudopatients.
A. Owen (1974)
10.1001/jama.1987.03400230065033
Evaluation of physical examination skills. Reliability of faculty observers and patient instructors.
D. Elliot (1987)
10.7326/0003-4819-114-5-393
Assessment of clinical skills of residents utilizing standardized patients. A follow-up study and recommendations for application.
P. Stillman (1991)
10.1037/h0026256
Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit.
J. Cohen (1968)
10.1111/j.1365-2923.1988.tb00761.x
Psychometric characteristics of the objective structured clinical examination.
D. Newble (1988)
10.7326/0003-4819-76-3-491
The new procedure for evaluating the clinical competence of candidates to be certified by the American Board of Internal Medicine.
R. Petersdorf (1972)
10.1097/00001888-197810000-00006
The use of instructor-patients to teach physical examination techniques.
K. K. Anderson (1978)
10.1097/00001888-197412000-00009
Utilization of simulated patients to teach the routine pelvic examination.
T. R. Godkins (1974)
Generalizability theory: a review. In: Traub RE (ed.). New directions for testing and measurement (no
RL Brennan (1979)
Intraclass correlations in assessing rater reliability
PE Shrout (1979)
10.1001/archinte.1985.00360060189029
A comparative trial of the clinical evaluation exercise.
F. Kroboth (1985)
THE PROGRAMMED PATIENT: A TECHNIQUE FOR APPRAISING STUDENT PERFORMANCE IN CLINICAL NEUROLOGY.
H. Barrows (1964)
The medical interview and related skills
M. Lipkin (1987)
10.1097/00001888-196602000-00006
Direct observation as a means of teaching and evaluating clinical skills
Hinz Cf (1966)
10.1036/1097-8542.498010
Performance rating.
J. Noak (1999)
10.1007/BF02600890
Examining the ABIM’s evaluation form
John A. Benson (2007)
10.1136/bmj.1.5955.447
Assessment of clinical competence using objective structured examination.
R. Harden (1975)
Generalizability theory : a review
RL Brennan
10.7326/0003-4819-105-5-762
Assessing clinical skills of residents with standardized patients.
P. Stillman (1986)
10.1097/00001888-198405000-00006
A survey of clinical skills evaluation practices in internal medicine residency programs.
L. Blank (1984)
10.1016/0002-9343(87)90494-3
An objective measure of clinical performance.
E. Petrusa (1987)
Evaluating evaluation : assessment of the American Board of Internal Medicine resident evaluation form
WG Thompson (1990)
10.1007/BF02599524
How accurate are faculty evaluations of clinical competence?
J. Herbers (2007)
10.1097/00001888-198410000-00004
Clinical competence certification: a critical appraisal.
J. Woolliscroft (1984)



This paper is referenced by
Assessment of clinical competence: reliability, validity, feasibility and educational impact of the mini-CEX
Araken Alves de Lima (2013)
10.1287/mnsc.2017.2883
Doing Well by Making Well: The Impact of Corporate Wellness Programs on Employee Productivity
Timothy Gubler (2018)
10.1080/01421590701352261
Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training
Alberto Alves de Lima (2007)
Assessment of communication skills in family medicine
Douglas Klein (2015)
10.4300/JGME-D-09-00090.1
A simplified observation tool for residents in the outpatient clinic.
J. Gigante (2010)
10.1097/00001888-200610001-00019
The Professionalism Mini-Evaluation Exercise: A Preliminary Investigation
R. Cruess (2006)
10.1515/dx-2019-0089
Assessing physical examination skills using direct observation and volunteer patients.
B. Clark (2020)
Evaluating Student Performance in the Experiential Setting with Confidence.
D. E. Beck (1995)
10.1007/S12507-010-0129-7
Achtergronden bij de Richtlijn korte praktijkbeoordeling
Th. J. ten Cate (2010)
10.1046/j.1525-1497.2001.016007427.x
Effectiveness of a focused educational intervention on resident evaluations from faculty
E. Holmboe (2007)
10.1016/j.sdentj.2017.07.001
Reliability of rubrics in the assessment of orthodontic oral presentation
N. Bindayel (2017)
10.1001/jama.2009.1365
Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review.
J. Kogan (2009)
10.1177/1477750917724332
The applicability of official professionalism recommendations in the emergency department
Amirhosein Jahanshir (2017)
10.1192/PB.BP.107.016576
Is CEX good for psychiatry? An evaluation of workplace-based assessment
G. Searle (2008)
10.1007/BF02599606
Didactic value of the clinical evaluation exercise missed opportunities
F. Kroboth (2007)
Inter-Rater Reliability Of Objective Structured Long Examination Record.
Syed Inamullah Shah (2018)
10.1007/s13187-017-1269-5
Assessing Communication Skills in Real Medical Encounters in Oncology: Development and Validation of the ComOn-Coaching Rating Scales
Marcelo Niglio de Figueiredo (2017)
10.1016/j.rce.2012.04.004
[The Computer Book of the Internal Medicine resident: validity and reliability of a questionnaire for self-assessment of competences in internal medicine residents].
Joaquim Oristrell (2012)
10.1111/j.1365-2923.2011.04075.x
A systematic review of the reliability of objective structured clinical examination scores.
M. Brannick (2011)
10.1111/j.1525-1497.2005.0140.x
A randomized-controlled study of encounter cards to improve oral case presentation skills of medical students
Sarang Kim (2005)
THE MINI MILESTONES ASSESSMENT (MINI-MAS), A DIRECT OBSERVATIONAL TOOL TO ASSESS CLINICAL MILESTONES IN THE ERA OF COMPETENCY-BASED EDUCATION
Moyez Ladhani (2015)
10.1136/emermed-2012-201237
A multicentre observational study to evaluate a new tool to assess emergency physicians' non-technical skills
Lynsey Flowerdew (2012)
10.1007/s11606-006-5083-8
The Impact of Feedback to Medical Housestaff on Chart Documentation and Quality of Care in the Outpatient Setting
D. Opila (2006)
Developing Oral Case Presentation Skills: Peer and Self-Evaluations as Instructional Tools.
D. Williams (2016)
10.1080/01421590902887404
Assessment of medical professionalism: Who, what, when, where, how, and … why?
Richard E. Hawkins (2009)
10.2196/14658
A Virtual Counseling Application Using Artificial Intelligence for Communication Skills Training in Nursing Education: Development Study
S. Shorey (2019)
10.1007/s11606-008-0842-3
Effect of Rater Training on Reliability and Accuracy of Mini-CEX Scores: A Randomized, Controlled Trial
D. Cook (2008)
10.2139/ssrn.2811785
Doing Well by Making Well: The Impact of Corporate Wellness Programs on Employee Productivity
Timothy Gubler (2017)
A Guide to Assessment Methods in Veterinary Medicine
S. Baillie (2008)
10.5959/EIMJ.V3I1.88
Faculty Development: Mini-CEX as Workplace-Based Assessment
Shahid Hassan (2011)
10.1192/PB.BP.113.043588
The long case: a case for revival?
Albert Michael (2013)
10.1080/10401330701366390
Decreasing Variability in Faculty Ratings of Student Case Presentations: A Faculty Development Intervention Focusing on Reflective Practice
M. Ottolini (2007)
See more
Semantic Scholar Logo Some data provided by SemanticScholar