This Article  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citing articles on HighWire  • Citing articles on Web of Science (1)  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Topic Collections  • Medical Practice  • Medical Education  • Alert me on articles by topic  Social Bookmarking   What's this?

Lasya Gaur; Susan Skochelak, MD, MPH
University of Wisconsin Medical School, Madison

JAMA. 2004;291:2143.

Tools to evaluate medical students serve a 2-fold purpose: they are expected to provide an accurate assessment of the clinical abilities of future physicians and to measure the successes and failures of curricular innovations in medical education. Results of educational research are often reported using scores from written examinations such as the US Medical Licensing Examinations (USMLE), faculty evaluations of clerkships, and the Objective Structured Clinical Examination (OSCE). While numerous barriers to evidence-based research in medical education have been identified,¹ another challenge is the discrepancy among results from these different types of evaluations.

The OSCE was developed to assess students' clinical skills in a standardized clinical setting using multiple stations and standardized patients. This test was first described in 1975 to assess clinical competence free from the biases of traditional evaluation methods.² Students usually spend a predetermined amount of time at each clinical station and take a focused history, provide differential diagnoses, and interpret test results. A study of all 126 US medical schools found that the prevalence of OSCE use has increased from 49 schools in 1997-1998 to 82 schools in the 2002-2003 academic year.³ This evaluation method has recently gained national attention, as all graduating US medical students will be required to take a competency-based skills examination for licensure beginning in 2004.

Several studies, however, have found inconsistent relationships between scores obtained on OSCEs and those from faculty evaluations and standardized written examinations. Prislin et al⁴ found a correlation of only 0.26 between OSCE scores and faculty evaluations of students during a third-year clerkship, a correlation of 0.22 between OSCEs and the National Board of Medical Examiners subject examinations, and correlation of 0.22 between faculty evaluations and written examinations. Similarly, a recent study found a correlation of less than 0.23 between OSCE scores in the fourth year of medical school and evaluation of clinical skills by residency directors. Correlations between faculty evaluations and class rank as well as USMLE step scores were also low (r<0.26).⁵ Another study also found a similarly low correlation (r = 0.30) between fourth-year OSCE scores and first-year residency director evaluations in a family medicine clerkship.⁶

There have also been inconsistencies between results of studies comparing objective assessment tools themselves. Ayers and Boulet⁷ reported varying correlations between the different components of Clinical Skills Assessment examination for foreign medical students and scores on the USMLE steps 1 and 2 (r = 0.11-0.32 and r = 0.29-0.43, respectively) and, as in the aforementioned studies, similarly low correlation when comparing components of the clinical skills examination and various components of faculty evaluations. In comparing second-year medical students' OSCE scores to USMLE step 1 scores over 3 years, Simon et al⁸ found only moderate correlation (r = 0.41) between results from the 2 tests.

These studies suggest that there is no single standard with which to evaluate medical students and the results of curricular innovations in medicine. None of these tests can be discounted, however, as each may measure a different parameter of competence. It has been suggested that while the OSCE reflects clinical skills and data gathering, standardized tests measure factual knowledge independent of clinical skills.⁸ Faculty evaluations do not correlate highly with results from either test and thus may reflect other qualities, such as interpersonal skills. Due to this lack of concordance between these different types of evaluations, each alone is probably insufficient to evaluate student performance.

REFERENCES
1. Hatala R, Guyatt G. Evaluating the teaching of evidence-based medicine. JAMA. 2002;288:1110-1112. FREE FULL TEXT 2. Harden RM, Stevenson M, Wilson Downie W, Wilson GM. Assessment of clinical competence using objective structured examination. BMJ. 1975;1:447-451. FREE FULL TEXT 3. Barzansky B, Etzel SI. Educational programs in US medical schools, 2002-2003. JAMA. 2003;290:1190-1196. FREE FULL TEXT 4. Prislin MD, Fitzpatrick CF, Lie D, Giglio M, Radecki S, Lewis E. Use of an objective structured clinical examination in evaluating student performance. Fam Med. 1998;30:338-344. PUBMED 5. Kahn MJ, Merrill WW, Anderson DS, Szerlip HM. Residency program director evaluations do not correlate with performance on a required 4th-year objective structured clinical examination. Teach Learn Med. 2001;13:9-12. FULL TEXT | WEB OF SCIENCE | PUBMED 6. Campos-Outcalt D, Watkins A, Fulginiti J, Kutob R, Gordon P. Correlations of family medicine clerkship evaluations and objective structured clinical examination scores and residency director ratings. Fam Med. 1999;31:90-94. PUBMED 7. Ayers WR, Boulet JR. Establishing the validity of test score inferences: performance of 4th-year US medical students on the ECFMG clinical skills assessment. Teach Learn Med. 2001;13:214-220. PUBMED 8. Simon SR, Volkan K, Claus H, Duffey C, Fletcher SW. The relationship between second-year medical students' OSCE scores and USMLE step 1 scores. Med Teach. 2002;24:535-539. PUBMED

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Playing Doctor
Henry
JAMA 2005;294:2138-2140.
FULL TEXT