This Article  • Abstract  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (312)  • Contact me when this article is cited  Related Content  • Related articles  • Similar articles in JAMA  Topic Collections  • Quality of Care  • Quality of Care, Other  • Review  • Alert me on articles by topic  Social Bookmarking   What's this?

Ronald M. Epstein, MD; Edward M. Hundert, MD

JAMA. 2002;287:226-235.

ABSTRACT
Context  Current assessment formats for physicians and trainees reliably test core knowledge and basic skills. However, they may underemphasize some important domains of professional medical practice, including interpersonal skills, lifelong learning, professionalism, and integration of core knowledge into clinical practice.

Objectives  To propose a definition of professional competence, to review current means for assessing it, and to suggest new approaches to assessment.

Data Sources  We searched the MEDLINE database from 1966 to 2001 and reference lists of relevant articles for English-language studies of reliability or validity of measures of competence of physicians, medical students, and residents.

Study Selection  We excluded articles of a purely descriptive nature, duplicate reports, reviews, and opinions and position statements, which yielded 195 relevant citations.

Data Extraction  Data were abstracted by 1 of us (R.M.E.). Quality criteria for inclusion were broad, given the heterogeneity of interventions, complexity of outcome measures, and paucity of randomized or longitudinal study designs.

Data Synthesis  We generated an inclusive definition of competence: the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and the community being served. Aside from protecting the public and limiting access to advanced training, assessments should foster habits of learning and self-reflection and drive institutional change. Subjective, multiple-choice, and standardized patient assessments, although reliable, underemphasize important domains of professional competence: integration of knowledge and skills, context of care, information management, teamwork, health systems, and patient-physician relationships. Few assessments observe trainees in real-life situations, incorporate the perspectives of peers and patients, or use measures that predict clinical outcomes.

Conclusions  In addition to assessments of basic skills, new formats that assess clinical reasoning, expert judgment, management of ambiguity, professionalism, time management, learning strategies, and teamwork promise a multidimensional assessment while maintaining adequate reliability and validity. Institutional support, reflection, and mentoring must accompany the development of assessment programs.

INTRODUCTION

Jump to Section  • Top  • Introduction  • Defining professional competence  • Current means of assessment  • Methods  • Comment  • Author information  • References

Medical schools, postgraduate training programs, and licensing bodies conduct assessments to certify the competence of future practitioners, discriminate among candidates for advanced training, provide motivation and direction for learning, and judge the adequacy of training programs. Standards for professional competence delineate key technical, cognitive, and emotional aspects of practice, including those that may not be measurable.^1-2 However, there is no agreed-upon definition of competence that encompasses all important domains of professional medical practice. In response, the Accreditation Council for Graduate Medical Education defined 6 areas of competence and some means of assessing them³: patient care (including clinical reasoning), medical knowledge, practice-based learning and improvement (including information management), interpersonal and communication skills, professionalism, and systems-based practice (including health economics and teamwork).³

In this article, we will advance a definition of professional competence of physicians and trainees that expands on these 6 areas, perform an evidence-based critique of current methods of assessing these areas of competence, and propose new means for assessing residents and medical students.

DEFINING PROFESSIONAL COMPETENCE

Jump to Section  • Top  • Introduction  • Defining professional competence  • Current means of assessment  • Methods  • Comment  • Author information  • References

Building on prior definitions,^1-3 we propose that professional competence is the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served. Competence builds on a foundation of basic clinical skills, scientific knowledge, and moral development. It includes a cognitive function—acquiring and using knowledge to solve real-life problems; an integrative function—using biomedical and psychosocial data in clinical reasoning; a relational function—communicating effectively with patients and colleagues; and an affective/moral function—the willingness, patience, and emotional awareness to use these skills judiciously and humanely (BOX 1). Competence depends on habits of mind, including attentiveness, critical curiosity, self-awareness, and presence. Professional competence is developmental, impermanent, and context-dependent.

Box 1. Dimensions of Professional Competence Cognitive Core knowledge Basic communication skills Information management Applying knowledge to real-world situations Using tacit knowledge and personal experience Abstract problem-solving Self-directed acquisition of new knowledge Recognizing gaps in knowledge Generating questions Using resources (eg, published evidence, colleagues) Learning from experience Technical Physical examination skills Surgical/procedural skills Integrative Incorporating scientific, clinical, and humanistic judgment Using clinical reasoning strategies appropriately (hypothetico-deductive, pattern-recognition, elaborated knowledge) Linking basic and clinical knowledge across disciplines Managing uncertainty Context Clinical setting Use of time Relationship Communication skills Handling conflict Teamwork Teaching others (eg, patients, students, and colleagues) Affective/Moral Tolerance of ambiguity and anxiety Emotional intelligence Respect for patients Responsiveness to patients and society Caring Habits of Mind Observations of one's own thinking, emotions, and techniques Attentiveness Critical curiosity Recognition of and response to cognitive and emotional biases Willingness to acknowledge and correct errors RETURN TO TEXT

Acquisition and Use of Knowledge

Evidence-based medicine is an explicit means for generating an important answerable question, interpreting new knowledge, and judging how to apply that knowledge in a clinical setting.⁴ But Polanyi⁵ argues that competence is defined by tacit rather than explicit knowledge. Tacit knowledge is that which we know but normally do not explain easily, including the informed use of heuristics (rules of thumb), intuition, and pattern recognition. The assessment of evidence-based medicine skills is difficult because many of the heuristics used by novices are replaced by shortcuts in the hands of experts,⁶ as are other clinical skills.⁷

Personal knowledge is usable knowledge gained through experience.⁸ Clinicians use personal knowledge when they observe a patient's demeanor (such as a facial expression) and arrive at a provisional diagnosis (such as Parkinson disease) before eliciting the specific information to confirm it. Because experience does not necessarily lead to learning and competence,⁹ cognitive and emotional self-awareness is necessary to help physicians question, seek new information, and adjust for their own biases.

Integrative Aspects of Care

Professional competence is more than a demonstration of isolated competencies¹⁰; "when we see the whole, we see its parts differently than when we see them in isolation."¹¹ For example, the student who can elicit historical data and physical findings, who can suture well, who knows the anatomy of the gallbladder and the bile ducts, and who can draw the biosynthetic pathway of bilirubin may not accurately diagnose and manage a patient with symptomatic gallstones. A competent clinician possesses the integrative ability to think, feel, and act like a physician.^{6, 12-15} Schon¹⁶ argues that professional competence is more than factual knowledge and the ability to solve problems with clear-cut solutions: it is defined by the ability to manage ambiguous problems, tolerate uncertainty, and make decisions with limited information.

Competence depends on using expert scientific, clinical, and humanistic judgment to engage in clinical reasoning.^14-15,17-18 Although expert clinicians often use pattern recognition for routine problems¹⁹ and hypothetico-deductive reasoning for complex problems outside their areas of expertise, expert clinical reasoning usually involves working interpretations¹² that are elaborated into branching networks of concepts.^20-22 These networks help professionals initiate a process of problem solving from minimal information and use subsequent information to refine their understanding of the problem. Reflection allows practitioners to examine their own clinical reasoning strategies.

Building Therapeutic Relationships

The quality of the patient-physician relationship affects health and the recovery from illness,^23-24 costs,²⁵ and outcomes of chronic diseases^26-29 by altering patients' understanding of their illnesses and reducing patient anxiety.²⁶ Key measurable patient-centered²⁸ (or relationship-centered)^30-31 behaviors include responding to patients' emotions and participatory decision making.²⁹

Medical errors are often due to the failure of health systems rather than individual deficiencies.^32-34 Thus, the assessment of teamwork and institutional self-assessment might effectively complement individual assessments.

Affective and Moral Dimensions

Moral and affective domains of practice may be evaluated more accurately by patients and peers than by licensing bodies or superiors.³⁵ Only recently have validated measures captured some of the intangibles in medicine, such as trust^36-37 and professionalism.^38-39 Recent neurobiological research indicates that the emotions are central to all judgment and decision making,¹³ further emphasizing the importance of assessing emotional intelligence and self-awareness in clinical practice.^{1, 40-42}

Habits of Mind

Competence depends on habits of mind that allow the practitioner to be attentive, curious, self-aware, and willing to recognize and correct errors.⁴³ Many physicians would consider these habits of mind characteristic of good practice, but they are especially difficult to objectify. A competent physician, for example, should be able to judge his or her level of anxiety when facing an ambiguous clinical presentation and be aware of how the anxiety of uncertainty may be influencing his or her clinical judgment. Errors in medicine, according to this view, may result from overcertainty that one's impressions are beyond doubt.^{41, 43-44}

Context

Competence is context-dependent. Competence is a statement of relationship between an ability (in the person), a task (in the world),⁴⁵ and the ecology of the health systems and clinical contexts in which those tasks occur.^46-47 This view stands in contrast to an abstract set of attributes that the physician possesses—knowledge, skills, and attitudes—that are assumed to serve the physician well in all the situations that he or she encounters. For example, rather than assessing a student's competence in diagnosing and treating heart disease (a disease-specific domain) by dividing it into competencies (physical examination, interpretation of electrocardiogram, and pharmacology of -blockers), our view is that competence is defined by the interaction of the task (the concrete process of diagnosing and treating Mrs Brown, a 52-year-old business executive who is now in the emergency department because of new-onset chest pain), the clinician's abilities (eliciting information, forming a therapeutic relationship, performing diagnostic maneuvers, and making judgments about treatment), and the health system (good insurance and ready access to care). Caring for Mrs Brown requires different skills than caring for Ms Hall, a 52-year-old uninsured homeless woman who has similar symptoms and receives episodic care at an inner-city clinic.

Development

Competence is developmental. There is debate about which aspects of competence should be acquired at each stage of training. For example, early clinical experiences and problem-based learning formats encourage clinical reasoning skills formerly relegated to the final years of medical school. But students tend to use the same cognitive strategy for solving all problems, whereas experts draw on several strategies,⁶ which raises the question of whether assessment of practicing physicians should be qualitatively different from the assessment of a student. Determining how and at what level of training the patient-physician relationship should be assessed is also difficult. For example, participatory decision making correlates with clinical outcomes,^{25, 29} but it is unclear when trainees should be assessed on this skill. Although a third-year resident might be expected to counsel a fearful diabetic patient about the need to start insulin, a third-year student might be expected only to elicit the patient's preferences, emotions, and expectations. Changes in medical practice and the context of care invite redefinitions of competence; for example, the use of electronic communication media⁴⁸ and changes in patient expectations.^49-50

CURRENT MEANS OF ASSESSMENT

Jump to Section  • Top  • Introduction  • Defining professional competence  • Current means of assessment  • Methods  • Comment  • Author information  • References

Assessment must take into account what is assessed, how it is assessed, and the assessment's usefulness in fostering future learning. In discussing validity of measures of competence in an era when reliable assessments of core knowledge, abstract problem solving, and basic clinical skills have been developed,^{45, 51-56} we must now establish that they encompass the qualities that define a good physician: the cognitive, technical, integrative, contextual, relational, reflective, affective, and moral aspects of competence. We distinguish between expert opinion, intermediate outcomes, and the few studies that show associations between results of assessments and actual clinical performance.^57-60

We consider how the process of assessment might foster future learning. Too often, practitioners select educational programs that are unlikely to influence clinical practice.⁶¹ Good assessment is a form of learning and should provide guidance and support to address learning needs. Finally, we address concerns that the medical profession still lacks adequate accountability to the public⁶² and has not done enough to reduce medical errors.^{32, 63}

Within each domain of assessment, there are 4 levels at which a trainee might be assessed (Figure 1).⁶⁴ The knows level refers to the recall of facts, principles, and theories. The knows how level involves the ability to solve problems and describe procedures. The shows how level usually involves human (standardized patient), mechanical, or computer simulations that involve demonstration of skills in a controlled setting. The does level refers to observations of real practice. For each of these levels, the student can demonstrate the ability to imitate or replicate a protocol, apply principles in a familiar situation, adapt principles to new situations, and associate new knowledge with previously learned principles.⁶⁵

View larger version (102K): [in this window] [in a new window] Figure. A Framework for Assessment The grid is filled out according to the type of assessment conducted, ie, standardized patient or simulation, video, postencounter probe, essay, or computer exercises. Each category can be combined with a number designating a category such as the name of a patient, a type of computer exercise, or a team exercise.

METHODS

Jump to Section  • Top  • Introduction  • Defining professional competence  • Current means of assessment  • Methods  • Comment  • Author information  • References

Using the MEDLINE database for 1966 to 2001, we searched for articles that studied the reliability or validity of measures of clinical or professional competence of physicians, medical students, and residents. An initial search using the following Medical Subject Headings of the National Library of Medicine yielded 2266 references: educational measurement, patient simulation, clinical competence OR professional competence AND reproducibility of results, validity OR research, OR the text word reliability. This set was reduced by including any of 20 text words describing assessment techniques; we used words such as OSCE, oral examination, peer assessment, triple jump, essay, portfolio, and CEX (clinical evaluation exercise), yielding 430 references. Articles of a purely descriptive nature, reviews that offered no new data, and opinions and position statements were excluded, yielding 101 English-language references. We surveyed the first 200 of the 2165 references excluded and found none that met our search criteria. Quality criteria for inclusion were broad, given the small number of controlled trials of assessment interventions and the complexity of outcome measures. Because we knew that MEDLINE search strategies would not capture all relevant studies, we searched reference lists in the 101 articles, other review articles, and books and did additional literature searches using the key authors of recent reviews; we gathered 94 additional relevant references. Of the 195 references, 124 presented new data on assessment of physicians.

Summary of Studies

The 3 most commonly used assessment methods are subjective assessments by supervising clinicians, multiple-choice examinations to evaluate factual knowledge and abstract problem solving,⁶⁶ and standardized patient assessments of physical examination and technical and communication skills.^67-69 Although curricular designs increasingly integrate core knowledge and clinical skills, most assessment methods evaluate these domains in isolation. Few assessments use measures such as participatory decision making⁷⁰ that predict clinical outcomes in real practice. Few reliably assess clinical reasoning, systems-based care, technology, and the patient-physician relationship.^{3, 69} The literature makes important distinctions between criteria for licensing examinations and program-specific assessments with mixed formative and summative goals.

Evaluation of factual knowledge and problem-solving skills by using multiple-choice questions offers excellent reliability^71-75 and assesses some aspects of context and clinical reasoning. Scores on Canadian licensing examinations, which include standardized patient assessment and multiple-choice tests, correlated positively with subsequent appropriate prescribing, mammographic screening, and referrals,⁵⁸ and multiple-choice certification examination scores correlated with subsequent faculty⁷⁶ and peer⁷⁷ ratings. Many have questioned the validity of multiple-choice examinations, though.^78-81 For example, compared with Florida family physicians who are not board-certified, those who are have nearly twice the risk of being sued.⁸² Standardized test scores have been inversely correlated with empathy, responsibility, and tolerance.⁸³ Also, because of lack of expertise and resources, few medical school examinations can claim to achieve the high psychometric standards of the licensing boards.

The Objective Structured Clinical Examination (OSCE) is a timed multistation examination often using standardized patients (SPs) to simulate clinical scenarios. The roles are portrayed accurately^{56, 84} and simulations are convincing; the detection rate of unannounced SPs in community practice is less than 10%.^{57, 59, 85-89} Communication, physical examination, counseling, and technical skills can be rated reliably if there is a sufficiently large number of SP cases^{67, 90-100} and if criteria for competence are based on evidence.¹⁰¹ Although few cases are needed to assess straightforward skills, up to 27 cases may be necessary to assess interpersonal skills reliably in high-stakes examinations.^102-103 Although SPs' ratings usually correlate with those of real patients,¹⁰⁴ differences have been noted.^105-107

Defining pass/fail criteria for OSCEs has been complex.^{54, 108-111} There is debate about who should rate student performance in an OSCE.¹¹² Ratings by the SP are generally accurate⁵² but may be hampered by memory failure, whereas external raters, either physicians or other SPs, may be less attuned to affective aspects of the interview and significantly increase the cost of the examination.

Checklist scores completed by physician-examiners in some studies improve with expertise of the examinees¹¹³ and with the reputation of the training program.^{90, 114} But global rating scales of interpersonal skills may be more valid than behavioral checklists.^{7, 115-116} The OSCE scores may not correlate with multiple-choice examinations and academic grades,^{90, 100, 117} suggesting that these tools measure different skills. Clinicians may behave differently in examination settings than in real practice,^{106, 118} and short OSCE stations can risk fragmentation and trivialization of isolated elements of what should be a coherent whole.¹¹⁹ The OSCE also has low test reliability for measuring clinical ethics.¹²⁰

There are few validated strategies to assess actual clinical practice, or Miller's does level. Subjective evaluation by residents and attending physicians is the major form of assessment during residency and the clinical clerkships and often includes the tacit elements of professional competence otherwise overlooked by objective assessment instruments. Faculty ratings of humanism predicted patient satisfaction in one study.¹²¹ However, evaluators often do not observe trainees directly. They often have different standards^122-123 and are subject to halo effects¹²⁴ and racial and sex bias.^125-126 Because of interpatient variability and low interrater reliability, each trainee must be subject to multiple assessments for patterns to emerge. Standardized rating forms for direct observation of trainees^127-132 and structured oral examination formats have been developed in response to this criticism.^133-134

The Royal College of General Practitioners, dissatisfied with the capability of the OSCE to evaluate competence for the final professional licensing examination, developed a format in which candidates for certification present several best-case videotapes of their performance in real clinical settings to a trained examiner who uses specified criteria for evaluation.¹³⁵ Although the face validity of such a measure is high and the format is well accepted by physicians,¹³⁶ the number of cases that should be presented to achieve adequate reliability is unclear.^137-139

Profiling by managed-care databases is increasingly used as an evaluation measure of clinical competence. However, data abstraction is complex¹⁴⁰ and defining competence in terms of cost and value is difficult. The underlying assumptions driving such evaluation systems may not be explicit. For example, cost analyses may favor physicians caring for more highly educated patients.¹⁴¹

Peer ratings are accurate and reliable measures of physician performance.^{77, 142} Peers may be in the best position to evaluate professionalism; people often act differently when not under direct scrutiny.¹⁴³ Anonymous medical student peer assessments of professionalism have raised awareness of professional behavior, fostered further reflection, helped students identify specific mutable behaviors, and been well accepted by students.³⁵ Students should be assessed by at least 8 of their classmates. The composite results should be edited to protect the confidentiality of the raters.

Self-assessments have been used with some success in standardized patient exercises¹⁴⁴ and in programs that offer explicit training in the use of self-assessment instruments.¹⁴⁵ Among trainees who did not have such training, however, self-assessment was neither valid nor accurate. Rather, it was more closely linked to the trainee's psychological sense of self-efficacy and self-confidence than to appropriate criteria, even among bright and motivated individuals.

COMMENT

Jump to Section  • Top  • Introduction  • Defining professional competence  • Current means of assessment  • Methods  • Comment  • Author information  • References

Aside from the need to protect the public by denying graduation to those few trainees who are not expected to overcome their deficiencies, the outcomes of assessment should foster learning, inspire confidence in the learner, enhance the learner's ability to self-monitor, and drive institutional self-assessment and curricular change. Given the difficulty in validating tests of basic skills, it is not surprising that there is scant literature on the assessment of learning, professionalism, teamwork, and systems-based care or on the ability of assessment programs to drive curricular change or reduce medical errors.

Assessment serves personal, institutional, and societal goals (BOX 2). Distinctions between these goals often are blurred in practice. For example, formative feedback is intended to foster individual reflection and remediation¹⁴⁶ but may be perceived as having evaluative consequences. Summative evaluation is a powerful means for driving curricular content and what students learn. Assessment provides information to allow institutions to choose among candidates for advanced training. The public expects greater self-monitoring, communication, and teamwork from health care practitioners.¹⁴⁷ The decline of public trust in medicine may reflect a growing concern that physicians are not achieving these goals.³⁶

Box 2. Some Purposes of Assessment For the Trainee Provide useful feedback about individual strengths and weaknesses that guides future learning Foster habits of self-reflection and self-remediation Promote access to advanced training For the Curriculum Respond to lack of demonstrated competence (denial of promotion, mandated remediation) Certify achievement of curricular goals Foster course or curricular change Create curricular coherence Cross-validate other forms of assessment in the curriculum Establish standards of competence for trainees at different levels For the Institution Guide a process of institutional self-reflection and remediation Discriminate among candidates for further training or promotion Express institutional values by determining what is assessed and how assessment is conducted Develop shared educational values among a diverse community of educators Promote faculty development Provide data for educational research For the Public Certify competence of graduates RETURN TO TEXT

Assessment is also a statement of institutional values. Devoting valuable curricular time to peer assessment of professionalism, for example, can promote those values that are assessed while encouraging curricular coherence and faculty development, especially if there are corresponding efforts at the institution toward self-assessment and change.

Whereas performance is directly measurable, competence is an inferred quality.¹⁴⁸ Performance on a multiple-choice test may exceed competence, as in the case of a trainee with a photographic memory but poor clinical judgment. Conversely, competence may exceed test performance, as in the case of a trainee with severe test anxiety. Correlation with National Board scores and feedback on graduates' performance can be useful in validating some assessment instruments but should be done with caution. For example, efficiency is highly valued in residents but less so in medical students.

Future Directions

Medical schools in Canada, the United Kingdom, Australia, Spain, the Netherlands, and the United States have made commitments to developing innovative assessments of professional competence, some of which we describe. These assessments are increasingly multimodal and tailored to the goals and context in which they will be used. Large-scale licensure examinations must use computer-gradable formats, but comprehensive examinations using structured direct observation,¹⁰⁷ OSCE stations, real patient cases,¹⁰⁷ case-based questions,⁷⁹ peer assessments, and essay-type questions¹⁴⁹ are reliable as well. Proponents of the new formats argue that they provide more useful feedback and are more efficient at the medical school or residency level (BOX 1 and BOX 3) than traditional formats.^{81, 150} They target core knowledge and clinical skills in different contexts and at different levels of assessment. Because of their complexity, a matrix (Figure 1) can be useful to display the domains assessed.

Box 3. Innovations in Assessing Professional Competence Multimethod assessment Clinical reasoning in situations that involve clinical uncertainty Standardized patient exercises linked to postencounter probes of pathophysiology and clinical reasoning Exercises to assess use of the medical literature Long-station standardized patient exercises Simulated continuity Teamwork exercises Unannounced standardized patients in clinical settings Assessments by patients Peer assessment of professionalism Portfolios of videotapes Mentored self-assessment Remediation based on a learning plan RETURN TO TEXT

Comprehensive assessments link content across several formats. Postencounter probes immediately after SP exercises using oral, essay, or multiple-choice questions test pathophysiology and clinical reasoning in context.^151-152 Triple-jump exercises¹⁵²—consisting of a case presentation, an independent literature search, and then an oral or written postencounter examination—test the use and application of the medical literature. Validated measures of reflective thinking¹⁵³ have been developed that use patient vignettes followed by questions that require clinical judgment. These measures reflect students' capacity to organize and link information; also, they predict clinical reasoning ability 2 years later.¹⁵³ Combining formats appears to have added value with no loss in reliability.^{150, 154} Ongoing educational outcomes research will show whether composite formats help students learn how to learn more effectively, develop habits of mind that characterize exemplary practice,⁴³ and provide a more multidimensional picture of the examinee than the individual unlinked elements. Two examples of comprehensive assessment formats follow.

Genetics, Evidence-Based Medicine, Screening, and Communication

A student is instructed to perform a literature search about genetic screening test for Alzheimer disease in anticipation of an SP encounter later that day. Assessment instruments include a structured evaluation of the search strategy and a communication rating scale, completed by an SP, that assesses the clarity of the student's presentation and the student's ability to involve the patient in the decision-making process. Next, the student completes an essay about the ethics of genetic screening and the genetics of Alzheimer disease. This exercise assesses the student's communication skills, clinical reasoning, ability to acquire and use new knowledge, and contextualized use of knowledge of genetics, health economics, and medical ethics.

Cognitive and Affective Challenges of Clinical Uncertainty

A rating scale is used to assess a resident on her ability to agree on a plan of action with an SP who portrays an outpatient demanding a computed tomographic scan for headaches without neurological signs. In a postencounter exercise, the resident creates a rank-order differential diagnosis and then answers a series of script concordance^{153, 155} questions in which the examinee is presented hypothetical additional data (for example, numbness in the left hand) and then asked to judge how her diagnostic hypotheses or therapeutic actions would change. Failure to include a key diagnostic possibility or the overestimation or underestimation of probability are criteria for evaluation. The goal of the exercise is to demonstrate emotional intelligence⁴⁰ and self-awareness in the context of conflict and ambiguity. Similar observations might be made with trainees' video portfolios of real clinical encounters.

Well-functioning health systems are characterized by continuity, partnership between physicians and patients, teamwork between health care practitioners, and communication between health care settings.^156-157 The use of time in a continuity relationship can be assessed with a series of SP or real-patient exercises. To assess partnership, patient assessment, currently used to assess physicians in practice,¹⁵⁸ is being tested for students and residents.^159-160 These efforts are guided by data showing that patients' ratings of communication and satisfaction correlate well with biomedical outcomes,^{24, 29} emotional distress,¹⁶¹ health care use,²⁵ and malpractice litigation.¹⁶² Patient ratings also have the potential to validate other measures of competence.¹⁶³ Several institutions assess teamwork by using peer assessments. Others use sophisticated mannequins to simulate acute cardiovascular physiological derangements found in intensive care settings^164-169; trainees are graded on teamwork as well as individual problem solving, and statistical adjustments can account for team composition. Communication between health settings could be assessed at the student level, for example, by grading of their written referral letters.¹⁷⁰

Although it could be argued that licensing boards do not have the mandate to remediate examinees who perform poorly or modify educational curricula, medical schools and residency programs do. Tests that demonstrate students' strengths or weaknesses may not provide the student with the opportunity to reflect on actual behaviors and patterns of thought that should be changed. To foster reflection and action, some institutions require a learning plan in which trainees chart their learning needs, the means of achieving them, expected time of completion, and means of verification^{146, 171} as a required outcome of an assessment.

A strong mentoring system should accompany any comprehensive assessment program. An inadequate system for feedback, mentoring, and remediation will subvert even the most well-conceived and validated examination. Curricular change also can be guided by results of assessments but requires a parallel process of institutional reflection, feedback, and remediation.

These new assessment formats are feasible, and several institutions have invested significant time and resources to develop them. The promise that a more comprehensive assessment of professional competence might improve practice, change medical education, and reduce medical errors should be studied in controlled trials. The public's trust in the medical profession and the ability of medical practitioners to learn from mistakes depends on valid and reliable means of assessment. Medical educators, professional societies, and licensing boards should view professional competence more comprehensively to improve the process of assessment.

AUTHOR INFORMATION

Jump to Section  • Top  • Introduction  • Defining professional competence  • Current means of assessment  • Methods  • Comment  • Author information  • References

Author Contributions: Study concept and design, critical revision of the manuscript for important intellectual content, and administrative, technical, or material support: Epstein, Hundert.

Acquisition of data, analysis and interpretation of data, and drafting of the manuscript: Epstein.

Acknowledgment: We would like to express thanks to Francesc Borrell-Carrio, MD, Daniel Federman, MD, Brian Hodges, MD, Daniel Klass, MD, Larry Mauksch, CSW, Timothy Quill, MD, Andres Sciolla, MD, and Kevin Volkan, PhD, for their critical review of the manuscript. Also we would like to acknowledge Anthony LaDuca, PhD, and Albert Oriol-Bosch, MD, for their contributions to our formulation of professional competence.

Corresponding Author and Reprints: Ronald M. Epstein, MD, University of Rochester School of Medicine and Dentistry, 885 South Ave, Rochester, NY 14620 (e-mail: ronald_epstein{at}urmc.rochester.edu).

Author Affiliations: Departments of Family Medicine (Dr Epstein), Psychiatry (Drs Epstein and Hundert), and Medical Humanities (Dr Hundert), University of Rochester School of Medicine and Dentistry, Rochester, NY.

REFERENCES

Jump to Section  • Top  • Introduction  • Defining professional competence  • Current means of assessment  • Methods  • Comment  • Author information  • References

1. American Board of Internal Medicine. Guide to Evaluation of Residents in Internal Medicine. Philadelphia, Pa: American Board of Internal Medicine; 1999. 2. Norman GR. Defining competence: a methodological review. In: Neufeld VR, Norman GR, eds. Assessing Clinical Competence. New York, NY: Springer; 1985:15-35. 3. ACGME Outcome Project. Accreditation Council for Graduate Medical Education Web site. Available at: http://www.acgme.org. 2000. Accessed October 1, 2001. 4. Sackett DL. Evidence-Based Medicine: How to Practice and Teach EBM. New York, NY: Churchill Livingstone; 1997. 5. Polanyi M. Personal Knowledge: Towards a Post-Critical Philosophy. Chicago, Ill: University of Chicago Press; 1974. 6. Benner P. From Novice to Expert. Menlo Park, Calif: Addison-Wesley; 1984. 7. Hodges B, Regehr G, McNaughton N, Tiberius R, Hanson M. OSCE checklists do not capture increasing levels of expertise. Acad Med. 1999;74:1129-1134. ISI | PUBMED 8. Dewey J. Experience and Nature. 1932 ed. New York, NY: Dover; 1958. 9. Langer EJ. The Power of Mindful Learning. Reading, Mass: Perseus Books; 1997. 10. Eraut M. Learning professional processes: public knowledge and personal experience. In: Eraut M, ed. Developing Professional Knowledge and Competence. London, England: Falmer Press; 1994:100-122. 11. Polanyi M. The logic of tacit inference. In: Grene M, ed. Knowing and Being: Essays. Chicago, Ill: University of Chicago; 1969:123-158. 12. Gale J, Marsden P. Clinical problem solving: the beginning of the process. Med Educ. 1982;16:22-26. ISI | PUBMED 13. Damasio AR. Descartes' Error: Emotion, Reason, and the Human Brain. New York, NY: GP Putnam's Sons; 1994. 14. Mandin H, Jones A, Woloschuk W, Harasym P. Helping students learn to think like experts when solving clinical problems. Acad Med. 1997;72:173-179. ISI | PUBMED 15. Friedman MH, Connell KJ, Olthoff AJ, Sinacore JM, Bordage G. Medical student errors in making a diagnosis. Acad Med. 1998;73(suppl):S19-S21. 16. Schon DA. The Reflective Practitioner. New York, NY: Basic Books; 1983. 17. Feinstein AR. "Clinical Judgment" revisited: the distraction of quantitative models. Ann Intern Med. 1994;120:799-805. FREE FULL TEXT 18. Downie RS, Macnaughton J, Randall R. Clinical Judgement: Evidence in Practice. Oxford, England: Oxford University Press; 2000. 19. Dunn MM, Woolliscroft JO. Assessment of a pattern-recognition examination in a clinical clerkship. Acad Med. 1994;69:683-684. ISI | PUBMED 20. Schmidt HG, Norman GR, Boshuizen HP. A cognitive perspective on medical expertise: theory and implication. Acad Med. 1990;65:611-621. [published correction appears in Acad Med. 1992;67:287]. ISI 21. Bordage G, Zacks R. The structure of medical knowledge in the memories of medical students and general practitioners: categories and prototypes. Med Educ. 1984;18:406-416. ISI | PUBMED 22. Bordage G. Elaborated knowledge: a key to successful diagnostic thinking. Acad Med. 1994;69:883-885. ISI | PUBMED 23. Starfield B, Wray C, Hess K, Gross R, Birk PS, D'Lugoff BC. The influence of patient-practitioner agreement on outcome of care. Am J Public Health. 1981;71:127-131. FREE FULL TEXT 24. The Headache Study Group of the University of Western Ontario. Predictors of outcome in headache patients presenting to family physicians—a one year prospective study. Headache. 1986;26:285-294. FULL TEXT | ISI | PUBMED 25. Stewart M, Brown JB, Donner A, et al. The impact of patient-centered care on outcomes. J Fam Pract. 2000;49:796-804. ISI | PUBMED 26. Di Blasi Z, Harkness E, Ernst E, Georgiou A, Kleijnen J. Influence of context effects on health outcomes: a systematic review. Lancet. 2001;357:757-762. FULL TEXT | ISI | PUBMED 27. Stewart M. Effective physician-patient communication and health outcomes: a review. CMAJ. 1995;152:1423-1433. ABSTRACT 28. Stewart M, Brown JB, Boon H, Galajda J, Meredith L, Sangster M. Evidence on patient-doctor communication. Cancer Prev Control. 1999;3:25-30. PUBMED 29. Kaplan SH, Greenfield S, Ware JE Jr. Assessing the effects of physician-patient interactions on the outcomes of chronic disease. Med Care. 1989;27:S110-S127. [published correction appears in Med Care. 1989;27:679]. FULL TEXT 30. Tresolini C and the Pew-Fetzer Task Force. Health Professions Education and Relationship-Centered Care: Report of the Pew-Fetzer Task Force on Advancing Psychosocial Health Education. San Francisco, Calif: Pew Health Professions Commission; 1994. 31. Candib LM. Medicine and the Family: A Feminist Perspective. New York, NY: Basic Books; 1995. 32. Ternov S. The human side of medical mistakes. In: Spath PL, ed. Error Reduction in Healthcare: A Systems Approach to Improving Patient Safety. San Francisco, Calif: Jossey Bass; 1999:97-138. 33. Chopra V, Bovill JG, Spierdijk J, Koornneef F. Reported significant observations during anaesthesia: a prospective analysis over an 18-month period. Br J Anaesth. 1992;68:13-17. FREE FULL TEXT 34. Cooper JB, Newbower RS, Long CD, McPeek B. Preventable anesthesia mishaps: a study of human factors. Anesthesiology. 1978;49:399-406. FULL TEXT | ISI | PUBMED 35. Asch E, Saltzberg D, Kaiser S. Reinforcement of self-directed learning and the development of professional attitudes through peer- and self-assessment. Acad Med. 1998;73:575. PUBMED 36. Safran DG, Montgomery JE, Chang H, Murphy J, Rogers WH. Switching doctors: predictors of voluntary disenrollment from a primary physician's practice. J Fam Pract. 2001;50:130-136. ISI | PUBMED 37. Safran DG, Taira DA, Rogers WH, Kosinski M, Ware JE, Tarlov AR. Linking primary care performance to outcomes of care. J Fam Pract. 1998;47:213-220. ISI | PUBMED 38. American Board of Internal Medicine. Project Professionalism. Philadelphia, Pa: American Board of Internal Medicine; 1998. 39. Peabody F. The care of the patient. JAMA. 1927;88:877-882. FREE FULL TEXT 40. Goleman D. Emotional Intelligence. New York, NY: Bantam Books; 1995. 41. Novack DH, Suchman AL, Clark W, Epstein RM, Najberg E, Kaplan C. Calibrating the physician: personal awareness and effective patient care. JAMA. 1997;278:502-509. FREE FULL TEXT 42. Ramsey PG, Wenrich M, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269:1655-1660. FREE FULL TEXT 43. Epstein RM. Mindful practice. JAMA. 1999;282:833-839. FREE FULL TEXT 44. Novack DH, Epstein RM, Paulsen RH. Toward creating physician-healers: fostering medical students' self-awareness, personal growth, and well-being. Acad Med. 1999;74:516-520. ISI | PUBMED 45. Klass D. Reevaluation of clinical competency. Am J Phys Med Rehabil. 2000;79:481-486. FULL TEXT | ISI | PUBMED 46. LaDuca A, Taylor DD, Hill IK. The design of a new physician licensure examination. Eval Health Prof. 1984;7:115-140. FREE FULL TEXT 47. Kane M. Model-based practice analysis and test specifications. Appl Meas Educ. 1997;10:18. 48. Dugdale DC, Epstein R, Pantilat SZ. Time and the patient-physician relationship. J Gen Intern Med. 1999;14(suppl 1):S34-S40. 49. Emanuel EJ, Dubler NN. Preserving the physician-patient relationship in the era of managed care. JAMA. 1995;273:323-335. FREE FULL TEXT 50. Kravitz RL, Cope DW, Bhrany V, Leake B. Internal medicine patients' expectations for care during office visits. J Gen Intern Med. 1994;9:75-81. ISI | PUBMED 51. LaDuca A. Validation of professional licensure examinations: professions theory, test design, and construct validity. Eval Health Prof. 1994;17:178-197. FREE FULL TEXT 52. De Champlain AF, Margolis MJ, King A, Klass DJ. Standardized patients' accuracy in recording examinees' behaviors using checklists. Acad Med. 1997;72(suppl 1):S85-S87. 53. Kopelow ML, Schnabl GK, Hassard TH, et al. Assessing practicing physicians in two settings using standardized patients. Acad Med. 1992;67(suppl):S19-S21. 54. Margolis MJ, De Champlain AF, Klass DJ. Setting examination-level standards for a performance-based assessment of physicians' clinical skills. Acad Med. 1998;73(suppl):S114-S116. 55. Ross LP, Clauser BE, Margolis MJ, Orr NA, Klass DJ. An expert-judgment approach to setting standards for a standardized-patient examination. Acad Med. 1996;71(suppl):S4-S6. 56. Tamblyn RM, Klass DJ, Schnabl GK, Kopelow ML. The accuracy of standardized patient presentation. Med Educ. 1991;25:100-109. ISI | PUBMED 57. McLeod PJ, Tamblyn RM, Gayton D, et al. Use of standardized patients to assess between-physician variations in resource utilization. JAMA. 1997;278:1164-1168. FREE FULL TEXT 58. Tamblyn R, Abrahamowicz M, Brailovsky C, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998;280:989-996. FREE FULL TEXT 59. Tamblyn RM. Use of standardized patients in the assessment of medical practice. CMAJ. 1998;158:205-207. PUBMED 60. Abrahamowicz M, Tamblyn RM, Ramsay JO, Klass DK, Kopelow ML. Detecting and correcting for rater-induced differences in standardized patient tests of clinical competence. Acad Med. 1990;65(suppl):S25-S26. 61. Davis D. Does CME work? an analysis of the effect of educational activities on physician performance or health care outcomes. Int J Psychiatry Med. 1998;28:21-39. ISI | PUBMED 62. Kassirer JP. Pseudoaccountability. Ann Intern Med. 2001;134:587-590. FREE FULL TEXT 63. Kohn LT, Corrigan JM, Donaldson MS. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000. 64. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(suppl):S63-S67. 65. Broudy HS, Smith BO, Burnett J. Democracy and Excellence in American Secondary Education. Chicago, Ill: Rand McNally; 1964. 66. Newble D, Dawson B, Dauphinee D. Guidelines for assessing clinical competence. Teaching Learning Med. 1994;6:213-220. 67. Colliver JA, Swartz MH. Assessing clinical performance with standardized patients. JAMA. 1997;278:790-791. FULL TEXT | PUBMED 68. Barrows HS. An overview of the uses of standardized patients for teaching and evaluating clinical skills: AAMC. Acad Med. 1993;68:443-451. ISI | PUBMED 69. Kassebaum DG, Eaglen RH. Shortcomings in the evaluation of students' clinical skills and behaviors in medical school. Acad Med. 1999;74:842-849. ISI | PUBMED 70. Greenfield S, Kaplan S, Ware JE Jr. Expanding patient involvement in care: effects on patient outcomes. Ann Intern Med. 1985;102:520-528. FREE FULL TEXT 71. Case SM, Becker DF, Swanson DB. Relationship between scores on NBME basic science subject tests and the first administration of the newly designed NBME Part I examination. Acad Med. 1992;67(suppl):S13-S15. 72. Norcini JJ, Swanson DB, Grosso LJ, Shea JA, Webster GD. A comparison of knowledge, synthesis, and clinical judgment: multiple-choice questions in the assessment of physician competence. Eval Health Prof. 1984;7:485-499. FREE FULL TEXT 73. Norcini JJ, Swanson DB, Grosso LJ, Webster GD. Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence. Med Educ. 1985;19:238-247. ISI | PUBMED 74. Case SM, Ripkey DR, Swanson DB. The relationship between clinical science performance in 20 medical schools and performance on Step 2 of the USMLE licensing examination: 1994-95 Validity Study Group for USMLE Step 1 and 2 Pass/Fail Standards. Acad Med. 1996;71(suppl):S31-S33. 75. Case SM, Swanson DB. Validity of NBME Part I and Part II scores for selection of residents in orthopaedic surgery, dermatology, and preventive medicine. Acad Med. 1993;68(suppl):S51-S56. 76. Downing SM. The assessment of clinical competence on the emergency medicine specialty certification examination: the validity of clinically relevant multiple-choice items. Ann Emerg Med. 1980;9:554-556. FULL TEXT | ISI | PUBMED 77. Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA. 1993;269:1655-1660. FREE FULL TEXT 78. Holsgrove GJ. Guide to postgraduate exams: multiple-choice questions. Br J Hosp Med. 1992;48:757-761. ISI | PUBMED 79. Schuwirth LW, Verheggen MM, van der Vleuten C, Boshuizen HP, Dinant GJ. Do short cases elicit different thinking processes than factual knowledge questions do? Med Educ. 2001;35:348-356. FULL TEXT | ISI | PUBMED 80. Veloski JJ, Rabinowitz HK, Robeson MR, Young PR. Patients don't present with five choices: an alternative to multiple-choice tests in assessing physicians' competence. Acad Med. 1999;74:539-546. ISI | PUBMED 81. van der Vleuten C. Validity of final examinations in undergraduate medical training. BMJ. 2000;321:1217-1219. FREE FULL TEXT 82. Ely JW, Dawson JD, Young PR, et al. Malpractice claims against family physicians: are the best doctors sued more? J Fam Pract. 1999;48:23-30. ISI | PUBMED 83. Tutton PJ. Psychometric test results associated with high achievement in basic science components of a medical curriculum. Acad Med. 1996;71:181-186. ISI | PUBMED 84. Tamblyn RM, Klass DK, Schanbl GK, Kopelow ML. Factors associated with the accuracy of standardized patient presentation. Acad Med. 1990;65(suppl):S55-S56. 85. Tamblyn R, Berkson L, Dauphinee WD, et al. Unnecessary prescribing of NSAIDs and the management of NSAID-related gastropathy in medical practice. Ann Intern Med. 1997;127:429-438. FREE FULL TEXT 86. Carney PA, Allen JD, Eliassen MS, Owen M, Badger LW. Recognizing and managing depression in primary care: a standardized patient study. J Fam Pract. 1999;48:965-972. ISI | PUBMED 87. Carney PA, Dietrich AJ, Freeman DH Jr, Mott LA. The periodic health examination provided to asymptomatic older women: an assessment using standardized patients. Ann Intern Med. 1993;119:129-135. FREE FULL TEXT 88. Carney PA, Dietrich AJ, Freeman DH Jr, Mott LA. A standardized-patient assessment of a continuing medical education program to improve physicians' cancer-control clinical skills. Acad Med. 1995;70:52-58. ISI | PUBMED 89. Tamblyn RM, Klass DJ, Schnabl GK, Kopelow ML. The accuracy of standardized patient presentation. Med Educ. 1991;25:100-109. ISI | PUBMED 90. van der Vleuten C, Sawnson DB. Assessment of clinical skills with standardized patients: state of the art. Teaching Learning Med. 1990;2:58-76. 91. Grand'Maison P, Lescop J, Rainsberry P, Brailovsky CA. Large-scale use of an objective, structured clinical examination for licensing family physicians. CMAJ. 1992;146:1735-1740. ABSTRACT 92. Joorabchi B. Objective structured clinical examination in a pediatric residency program. AJDC. 1991;145:757-762. PUBMED 93. A-Latif A. An examination of the examinations: the reliability of the objective structured clinical examination and clinical examination. Med Teach. 1992;14:179-183. ISI | PUBMED 94. MacRae HM, Cohen R, Regehr G, Reznick R, Burnstein M. A new assessment tool: the patient assessment and management examination. Surgery. 1997;122:335-343. FULL TEXT | ISI | PUBMED 95. Matsell DG, Wolfish NM, Hsu E. Reliability and validity of the objective structured clinical examination in paediatrics. Med Educ. 1991;25:293-299. ISI | PUBMED 96. Vu NV, Barrows HS, Marcy ML, Verhulst SJ, Colliver JA, Travis T. Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine. Acad Med. 1992;67:42-50. ISI | PUBMED 97. Vu NV, Distlehorst LH, Verhulst SJ, Colliver JA. Clinical performance-based test sensitivity and specificity in predicting first-year residency performance. Acad Med. 1993;68(suppl):S41-S45. 98. Newble DI, Swanson DB. Psychometric characteristics of the objective structured clinical examination. Med Educ. 1988;22:325-334. ISI | PUBMED 99. Petrusa ER, Blackwell TA, Ainsworth MA. Reliability and validity of an objective structured clinical examination for assessing the clinical performance of residents. Arch Intern Med. 1990;150:573-577. FREE FULL TEXT 100. Gomez JM, Prieto L, Pujol R, et al. Clinical skills assessment with standardized patients. Med Educ. 1997;31:94-98. ISI | PUBMED 101. Gorter S, Rethans JJ, Scherpbier A, et al. Developing case-specific checklists for standardized-patient-based assessments in internal medicine: a review of the literature. Acad Med. 2000;75:1130-1137. ISI | PUBMED 102. Swartz MH, Colliver JA, Bardes CL, Charon R, Fried ED, Moroff S. Validating the standardized-patient assessment administered to medical students in the New York City Consortium. Acad Med. 1997;72:619-626. ISI | PUBMED 103. Carline JD, Paauw DS, Thiede KW, Ramsey PG. Factors affecting the reliability of ratings of students' clinical skills in a medicine clerkship. J Gen Intern Med. 1992;7:506-510. ISI | PUBMED 104. Sanson-Fisher RW, Poole AD. Simulated patients and the assessment of medical students' interpersonal skills. Med Educ. 1980;14:249-253. ISI | PUBMED 105. Tamblyn R, Abrahamowicz M, Schnarch B, et al. Can standardized patients predict real-patient satisfaction with the doctor-patient relationship? Teaching Learning Med. 1994;6:36-44. 106. Pieters HM, Touw-Otten FW, De Melker RA. Simulated patients in assessing consultation skills of trainees in general practice vocational training: a validity study. Med Educ. 1994;28:226-233. ISI | PUBMED 107. Wass V, Jones R, van der Vleuten C. Standardized or real patients to test clinical competence? the long case revisited. Med Educ. 2001;35:321-325. FULL TEXT | ISI | PUBMED 108. Cusimano MD, Rothman A, Keystone J. Defining standards of competent performance on an OSCE. Acad Med. 1998;73(suppl):S112-S113. 109. Kaufman DM, Mann KV, Muijtjens AM, van der Vleuten C. A comparison of standard-setting procedures for an OSCE in undergraduate medical education. Acad Med. 2000;75:267-271. ISI | PUBMED 110. Travis TA, Colliver JA, Robbs RS, et al. Validity of simple approach to scoring and standard setting for standardized-patient cases in an examination of clinical competence. Acad Med. 1996;71(suppl):S84-S86. 111. Norcini JJ, Stillman PL, Sutnick AI, et al. Scoring and standard setting with standardized patients. Eval Health Prof. 1993;16:322-332. FREE FULL TEXT 112. Martin JA, Reznick RK, Rothman A, Tamblyn RM, Regehr G. Who should rate candidates in an objective structured clinical examination? Acad Med. 1996;71:170-175. ISI | PUBMED 113. Blane CE, Calhoun JG. Objectively evaluating student case presentations. Invest Radiol. 1985;20:121-123. FULL TEXT | ISI | PUBMED 114. Stillman P, Swanson D, Regan MB, et al. Assessment of clinical skills of residents utilizing standardized patients: a follow-up study and recommendations for application. Ann Intern Med. 1991;114:393-401. FREE FULL TEXT 115. Cohen R, Rothman AI, Poldre P, Ross J. Validity and generalizability of global ratings in an objective structured clinical examination. Acad Med. 1991;66:545-548. ISI | PUBMED 116. Swartz MH, Colliver JA, Bardes CL, Charon R, Fried ED, Moroff S. Global ratings of videotaped performance versus global ratings of actions recorded on checklists: a criterion for performance assessment with standardized patients. Acad Med. 1999;74:1028-1032. ISI | PUBMED 117. Mumford E, Schlesinger H, Cuerdon T, Scully J. Ratings of videotaped simulated patient interviews and four other methods of evaluating a psychiatry clerkship. Am J Psychiatry. 1987;144:316-322. FREE FULL TEXT 118. Tamblyn RM, Abrahamowicz M, Berkson L, et al. First-visit bias in the measurement of clinical competence with standardized patients. Acad Med. 1992;67(suppl):S22-S24. 119. Ferrell BG, Thompson BL. Standardized patients: a long-station clinical examination format. Med Educ. 1993;27:376-381. ISI | PUBMED 120. Singer PA, Robb A, Cohen R, Norman G, Turnbull J. Performance-based assessment of clinical ethics using an objective structured clinical examination. Acad Med. 1996;71:495-498. ISI | PUBMED 121. McLeod PJ, Tamblyn R, Benaroya S, Snell L. Faculty ratings of resident humanism predict patient satisfaction ratings in ambulatory medical clinics. J Gen Intern Med. 1994;9:321-326. ISI | PUBMED 122. Irby D. Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med. 1995;70:898-931. ISI | PUBMED 123. Kalet A, Earp JA, Kowlowitz V. How well do faculty evaluate the interviewing skills of medical students? J Gen Intern Med. 1992;7:499-505. ISI | PUBMED 124. Rowland-Morin PA, Burchard KW, Garb JL, Coe NP. Influence of effective communication by surgery students on their oral examination scores. Acad Med. 1991;66:169-171. ISI | PUBMED 125. Wang-Cheng RM, Fulkerson PK, Barnas GP, Lawrence SL. Effect of student and preceptor gender on clinical grades in an ambulatory care clerkship. Acad Med. 1995;70:324-326. ISI | PUBMED 126. Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med. 1989;4:202-208. ISI | PUBMED 127. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123:795-799. FREE FULL TEXT 128. Dunnington GL, Wright K, Hoffman K. A pilot experience with competency-based clinical skills assessment in a surgical clerkship. Am J Surg. 1994;167:604-606. FULL TEXT | ISI | PUBMED 129. Dunnington G, Reisner L, Witzke D, Fulginiti J. Structured single-observer methods of evaluation for the assessment of ward performance on the surgical clerkship. Am J Surg. 1990;159:423-426. FULL TEXT | ISI | PUBMED 130. Keynan A, Friedman M, Benbassat J. Reliability of global rating scales in the assessment of clinical competence of medical students. Med Educ. 1987;21:477-481. ISI | PUBMED 131. Kroboth FJ, Hanusa BH, Parker S, et al. The inter-rater reliability and internal consistency of a clinical evaluation exercise. J Gen Intern Med. 1992;7:174-179. ISI | PUBMED 132. McKinley RK, Fraser RC, van der Vleuten C, Hastings AM. Formative assessment of the consultation performance of medical students in the setting of general practice using a modified version of the Leicester Assessment Package. Med Educ. 2000;34:573-579. FULL TEXT | ISI | PUBMED 133. Yang JC, Laube DW. Improvement of reliability of an oral examination by a structured evaluation instrument. J Med Educ. 1983;58:864-872. ISI | PUBMED 134. Stillman RM, Lane KM, Beeth S, Jaffe BM. Evaluation of the student: improving validity of the oral examination. Surgery. 1983;93:439-442. ISI | PUBMED 135. Lockie C. The Examination for Membership in the Royal College of General Practitioners. London, England: Royal College of General Practitioners; 1990. Occasional Paper 46. 136. Ram P, van der Vleuten C, Rethans JJ, Grol R, Aretz K. Assessment of practicing family physicians: comparison of observation in a multiple-station examination using standardized patients with observation of consultations in daily practice. Acad Med. 1999;74:62-69. ISI | PUBMED 137. Ram P, van der Vleuten C, Rethans JJ, Schouten B, Hobma S, Grol R. Assessment in general practice: the predictive value of written-knowledge tests and a multiple-station examination for actual medical performance in daily practice. Med Educ. 1999;33:197-203. FULL TEXT | ISI | PUBMED 138. Ram P, Grol R, Rethans JJ, Schouten B, van der Vleuten C, Kester A. Assessment of general practitioners by video observation of communicative and medical performance in daily practice: issues of validity, reliability and feasibility. Med Educ. 1999;33:447-454. FULL TEXT | ISI | PUBMED 139. Tate P, Foulkes J, Neighbour R, Campion P, Field S. Assessing physicians' interpersonal skills via videotaped encounters: a new approach for the Royal College of General Practitioners Membership examination. J Health Commun. 1999;4:143-152. FULL TEXT | ISI | PUBMED 140. Luck J, Peabody JW, Dresselhaus TR, Lee M, Glassman P. How well does chart abstraction measure quality? a prospective comparison of standardized patients with the medical record. Am J Med. 2000;108:642-649. FULL TEXT | ISI | PUBMED 141. Fiscella K, Franks P. Influence of patient education on profiles of physician practices. Ann Intern Med. 1999;131:745-751. FREE FULL TEXT 142. Tornow W, London M. Maximizing the Value of 360-Degree Feedback: A Process for Successful Individual and Organizational Development. San Francisco, Calif: Jossey-Bass Inc; 1998. 143. Hundert EM, Hafferty F, Christakis D. Characteristics of the informal curriculum and trainees' ethical choices. Acad Med. 1996;71:624-642. ISI | PUBMED 144. Kaiser S, Bauer JJ. Checklist self-evaluation in a standardized patient exercise. Am J Surg. 1995;169:418-420. FULL TEXT | ISI | PUBMED 145. Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med. 1991;66:762-769. ISI | PUBMED 146. Westburg J, Jason H. Collaborative Clinical Education: The Foundation of Effective Patient Care. New York, NY: Springer Publishing Co; 1993. 147. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001. 148. Gonzi A. The Development of Competence-Based Assessment Strategies for the Professions. Canberra: Australian Government Publishing Service; 1993. National Office of Everseas Skills Recognition Research Paper # 8. 149. Wass V, McGibbon D, van der Vleuten C. Composite undergraduate clinical examinations: how should the components be combined to maximize reliability? Med Educ. 2001;35:326-330. FULL TEXT | ISI | PUBMED 150. van der Vleuten CCP, Norman GR, De Graaff E. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991;25:110-118. ISI | PUBMED 151. Hodges B. Assessment Across Basic Science and Clinical Skills: Using OSCEs With Postencounter Basic Science Probes. 2001. Manual located at the University of Toronto, Toronto, Ontario. 152. Smith RM. The triple-jump examination as an assessment tool in the problem-based medical curriculum at the University of Hawaii. Acad Med. 1993;68:366-372. ISI | PUBMED 153. Brailovsky C, Charlin B, Beausoleil S, Cote S, van der Vleuten C. Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test. Med Educ. 2001;35:430-436. FULL TEXT | ISI | PUBMED 154. Verhoeven BH, Hamers JG, Scherpbier AJ, Hoogenboom RJ, van der Vleuten C. The effect on reliability of adding a separate written assessment component to an objective structured clinical examination. Med Educ. 2000;34:525-529. FULL TEXT | ISI | PUBMED 155. Charlin B, Roy L, Brailovsky C, Goulet F, van der Vleuten C. The Script Concordance test: a tool to assess the reflective clinician. Teach Learn Med. 2000;12:189-195. FULL TEXT | ISI | PUBMED 156. Risser DT, Simon R, Rice MM, Salisbury ML. A structured teamwork system to reduce clinical errors. In: Spath PL, ed. Error Reduction in Health Care: A Systems Approach to Improving Patient Safety. New York, NY: John Wiley & Sons; 1999:235-277. 157. Suchman AL, Botelho RJ, Hinton-Walker P. Partnerships in Healthcare: Transforming Relational Process. Rochester, NY: University of Rochester Press; 1998. 158. American Board of Internal Medicine. Patient and Physician Peer Assessment and the ABIM Program of Continuous Professional Development. Philadelphia, Pa: American Board of Internal Medicine; 2001. 159. Dauphinee WD. Assessing clinical performance. Where do we stand and what might we expect? JAMA. 1995;274:741-743. FREE FULL TEXT 160. Violato C, Marini A, Toews J, Lockyer J, Fidler H. Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians. Acad Med. 1997;72(suppl 1):S82-S84. 161. Roter DL, Hall JA, Kern DE, Barker LR, Cole KA, Roca RP. Improving physicians' interviewing skills and reducing patients' emotional distress—a randomized clinical trial. Arch Intern Med. 1995;155:1877-1884. FREE FULL TEXT 162. Beckman HB, Markakis KM, Suchman AL, Frankel RM. The doctor-patient relationship and malpractice: lessons from plaintiff depositions. Arch Intern Med. 1994;154:1365-1370. FREE FULL TEXT 163. Tamblyn R, Benaroya S, Snell L, McLeod P, Schnarch B, Abrahamowicz M. The feasibility and value of using patient satisfaction ratings to evaluate internal medicine residents. J Gen Intern Med. 1994;9:146-152. ISI | PUBMED 164. Helmreich RL. On error management: lessons from aviation. BMJ. 2000;320:781-785. FREE FULL TEXT 165. Helmreich RL, Davies JM. Anaesthetic simulation and lessons to be learned from aviation. Can J Anaesth. 1997;44:907-912. ISI | PUBMED 166. Schaefer HG, Helmreich RL. The importance of human factors in the operating room. Anesthesiology. 1994;80:479. FULL TEXT | ISI | PUBMED 167. Davies JM, Helmreich RL. Virtual reality in medical training. CMAJ. 1997;157:1352-1353. PUBMED 168. Helmreich RL, Musson DM. Surgery as team endeavour. Can J Anaesth. 2000;47:391-392. ISI | PUBMED 169. Sexton JB, Thomas EJ, Helmreich RL. Error, stress, and teamwork in medicine and aviation: cross sectional surveys. BMJ. 2000;320:745-749. FREE FULL TEXT 170. Epstein RM. Communication between primary care physicians and consultants. In: Suchman AL, Botelho RJ, Hinton-Walker P, eds. Partnerships in Health Care: Transforming Relational Process. Rochester, NY: University of Rochester Press; 1998:171-183. 171. Frankford DM, Konrad TR. Responsive medical professionalism: integrating education, practice, and community in a market-driven era. Acad Med. 1998;73:138-145. ISI | PUBMED

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati     What's this?

RELATED ARTICLES

Competence Is a Habit
David C. Leach
JAMA. 2002;287(2):243-244.
EXTRACT | FULL TEXT  

January 9, 2002
JAMA. 2002;287(2):255-256.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

The Silent Dimension: Expressing Humanism in Each Medical Encounter
Schattner
Arch Intern Med 2009;169:1095-1099.
FULL TEXT  

A Multisource and Repeated Measure Approach to Assessing Patient--Physician Relationship and Patient Satisfaction
Weng
Eval Health Prof 2009;32:128-143.
ABSTRACT  

Introduction: Effectiveness of Continuing Medical Education: American College of Chest Physicians Evidence-Based Educational Guidelines
Moores et al.
Chest 2009;135:5S-7S.
FULL TEXT  

Education Research: Assessment of neurology resident clinical competencies in the neurology clinic
Davis et al.
Neurology 2009;72:e1-e3.
ABSTRACT | FULL TEXT  

CPD and recertification: improving patient outcomes through focused learning
Bamrah and Bhugra
Adv. Psychiatr. Treat. 2009;15:2-6.
ABSTRACT | FULL TEXT  

Assessing Dental Students' Competence: Best Practice Recommendations in the Performance Assessment Literature and Investigation of Current Practices in Predoctoral Dental Education
Albino et al.
J Dent Educ 2008;72:1405-1435.
ABSTRACT | FULL TEXT  

How Many Unjustifiable Lectures Are Worth $2.4 Billion?
Dunn and Armstrong
Chest 2008;134:1117-1120.
FULL TEXT  

Effect of Lean method implementation in the histopathology section of an anatomical pathology laboratory
Raab et al.
J. Clin. Pathol. 2008;61:1193-1199.
ABSTRACT | FULL TEXT  

Mindfulness and Professionalism in Dentistry
Lovas et al.
J Dent Educ 2008;72:998-1009.
ABSTRACT | FULL TEXT  

Accreditation Council for Graduate Medical Education Initiatives Improve the Education of Cardiology Fellows
Kuvin
Circulation 2008;118:525-531.
FULL TEXT  

The case for resurrecting the long case
Teoh and Bowden
BMJ 2008;336:1250-1250.
FULL TEXT  

Evaluation of the assessment and grading of medical students on a neurology clerkship
Schmahmann et al.
Neurology 2008;70:706-712.
ABSTRACT | FULL TEXT  

The effects of problem-based learning during medical school on physician competency: a systematic review
Koh et al.
CMAJ 2008;178:34-41.
ABSTRACT | FULL TEXT  

Unexpected Intraoperative Patient Death: The Imperatives of Family- and Surgeon-Centered Care
Taylor et al.
Arch Surg 2008;143:87-92.
ABSTRACT | FULL TEXT  

Multicenter Implementation of a Shared Graduate Medical Education Resource
Sisson et al.
Arch Intern Med 2007;167:2476-2480.
ABSTRACT | FULL TEXT  

Faculty Development to Support Curriculum Change and Ensure the Future Vitality of Dental Education
Licari
J Dent Educ 2007;71:1509-1512.
ABSTRACT | FULL TEXT  

The Bits and Bytes to Advanced Graduate Degree Programs
Radler and Touger-Decker
Nutr Clin Pract 2007;22:647-652.
ABSTRACT | FULL TEXT  

Professionalism in Pediatrics
Fallat et al.
Pediatrics 2007;120:e1123-e1133.
ABSTRACT | FULL TEXT  

Emotional intelligence and patient-centred care
Birks and Watt
JRSM 2007;100:368-374.
ABSTRACT | FULL TEXT  

Preparing the Personal Physician for Practice (P4): Redesigning Family Medicine Residencies: New Wine, New Wineskins, Learning, Unlearning, and a Journey to Authenticity
Leach and Batalden
J Am Board Fam Med 2007;20:342-347.
ABSTRACT | FULL TEXT  

Clinical Instructors' Perceptions of Behaviors That Comprise Entry-Level Clinical Performance in Physical Therapist Students: A Qualitative Study
Jette et al.
ptjournal 2007;87:833-843.
ABSTRACT | FULL TEXT  

Ways of interacting: The standardization of communication in medical training
Islam and Zyphur
Human Relations 2007;60:769-792.
ABSTRACT  

Educating to Improve Patient Care: Integrating Quality Improvement Into a Medical School Curriculum
Varkey
American Journal of Medical Quality 2007;22:112-116.
ABSTRACT  

Ratings of Physician Communication by Real and Standardized Patients
Fiscella et al.
Ann Fam Med 2007;5:151-158.
ABSTRACT | FULL TEXT  

Pediatric and Emergency Medicine Residents' Attitudes and Practices for Analgesia and Sedation During Lumbar Puncture in Pediatric Patients
Breakey et al.
Pediatrics 2007;119:e631-e636.
ABSTRACT | FULL TEXT  

Emotional Intelligence and Perceived Stress in Dental Undergraduates: A Multinational Survey
Pau et al.
J Dent Educ 2007;71:197-204.
ABSTRACT | FULL TEXT  

Assessment in Medical Education
Epstein
NEJM 2007;356:387-396.
FULL TEXT  

Effect of Multisource Feedback on Resident Communication Skills and Professionalism: A Randomized Controlled Trial
Brinkman et al.
Arch Pediatr Adolesc Med 2007;161:44-49.
ABSTRACT | FULL TEXT  

Clinical Habits and the Psychiatrist: An Adult Developmental Model Focusing on the Academic Psychiatrist
Bourgeois and Servis
Acad. Psychiatry 2006;30:365-371.
ABSTRACT | FULL TEXT  

Evaluation of Resident Communication Skills and Professionalism: A Matter of Perspective?
Brinkman et al.
Pediatrics 2006;118:1371-1379.
ABSTRACT | FULL TEXT  

American Medical Education 100 Years after the Flexner Report.
Cooke et al.
NEJM 2006;355:1339-1344.
FULL TEXT  

Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
Davis et al.
JAMA 2006;296:1094-1102.
ABSTRACT | FULL TEXT  

Assessing the ACGME Competencies in Psychiatry Training Programs
Mennin
Acad. Psychiatry 2006;30:279-280.
FULL TEXT  

Assessing the ACGME Competencies in Psychiatry Training Programs
Swick et al.
Acad. Psychiatry 2006;30:330-351.
ABSTRACT | FULL TEXT  

Outcome of subacute low back pain: influence of patients' and rheumatologists' characteristics
Poiraudeau et al.
Rheumatology (Oxford) 2006;45:718-723.
ABSTRACT | FULL TEXT  

How can nutrition education contribute to competency-based resident evaluation?
Deen
Am. J. Clin. Nutr. 2006;83:976S-980S.
ABSTRACT | FULL TEXT  

Establishing a Program to Promote Professionalism and Effective Communication in Radiology
Donnelly and Strife
Radiology 2006;238:773-779.
ABSTRACT | FULL TEXT  

Incorporating simulation-based objective structured clinical examination into the israeli national board examination in anesthesiology.
Berkenstadt et al.
Anesth. Analg. 2006;102:853-858.
ABSTRACT | FULL TEXT  

Problem Doctors: Is There a System-Level Solution?
Leape and Fromson
ANN INTERN MED 2006;144:107-115.
ABSTRACT | FULL TEXT  

Teaching Non-Technical (Professional) Competence in a Veterinary School Curriculum
Burns et al.
jvme 2006;33:301-308.
ABSTRACT | FULL TEXT  

Primary care guidelines in rheumatology
Wise et al.
Rheumatology (Oxford) 2006;45:6-8.
FULL TEXT  

Disciplinary Action by Medical Boards and Prior Behavior in Medical School
Papadakis et al.
NEJM 2005;353:2673-2682.
ABSTRACT | FULL TEXT  

Medical Student Distress: Causes, Consequences, and Proposed Solutions
Dyrbye et al.
Mayo Clin Proc. 2005;80:1613-1622.
ABSTRACT  

A Simulation-Based Acute Skills Performance Assessment for Anesthesia Training
Murray et al.
Anesth. Analg. 2005;101:1127-1134.
ABSTRACT | FULL TEXT  

Competence in prehospital care: evolving concepts
Clements and Mackenzie
Emerg. Med. J. 2005;22:516-519.
ABSTRACT | FULL TEXT  

Quality and Quality Improvement in Forensic Mental Health Evaluations
Wettstein
J Am Acad Psychiatry Law 2005;33:158-175.
ABSTRACT | FULL TEXT  

Performance-Based Assessment of Radiology Faculty: A Practical Plan to Promote Improvement and Meet JCAHO Standards
Donnelly and Strife
Am. J. Roentgenol. 2005;184:1398-1401.
ABSTRACT | FULL TEXT  

Achieving Consensus on Competency in Community Pediatrics
Garfunkel et al.
Pediatrics 2005;115:1167-1171.
FULL TEXT  

Clinical Empathy as Emotional Labor in the Patient-Physician Relationship
Larson and Yao
JAMA 2005;293:1100-1106.
ABSTRACT | FULL TEXT  

Women in Academic Psychiatry
Bickel
Acad. Psychiatry 2004;28:285-291.
FULL TEXT  

Rheumatology curriculum: passport to the future successful handling of the musculoskeletal burden?
Goh et al.
Rheumatology (Oxford) 2004;43:1468-1472.
FULL TEXT  

The Positive Role of Professionalism and Ethics Training in Medical Education: A Comparison of Medical Student and Resident Perspectives
Roberts et al.
Acad. Psychiatry 2004;28:170-182.
ABSTRACT | FULL TEXT  

The Development and Application of an Instrument for Assessing Resident Competence During Preanesthesia Consultation
de Oliveira Filho and Schonhorst
Anesth. Analg. 2004;99:62-69.
ABSTRACT | FULL TEXT  

A New Model for Accreditation of Residency Programs in Internal Medicine
Goroll et al.
ANN INTERN MED 2004;140:902-909.
ABSTRACT | FULL TEXT  

Characteristics Associated With Physician Discipline: A Case-Control Study
Kohatsu et al.
Arch Intern Med 2004;164:653-658.
ABSTRACT | FULL TEXT  

Assessment of Competency in Child and Adolescent Psychiatry Training
Sargent et al.
Acad. Psychiatry 2004;28:18-26.
ABSTRACT | FULL TEXT  

Task Force Report 3. Report of the Task Force on Continuous Personal, Professional, and Practice Development in Family Medicine
Jones et al.
Ann Fam Med 2004;2:S65-S74.
ABSTRACT | FULL TEXT  

Assessing clinical competency in medical senior house officers: how and why should we do it?
Carr
Postgrad. Med. J. 2004;80:63-66.
ABSTRACT | FULL TEXT  

A pilot study in ophthalmology of inter-rater reliability in classifying diagnostic errors: an underinvestigated area of medical error
Margo
Qual Saf Health Care 2003;12:416-420.
ABSTRACT | FULL TEXT  

Effect of Communications Training on Medical Student Performance
Yedidia et al.
JAMA 2003;290:1157-1165.
ABSTRACT | FULL TEXT  

Physician Clinical Performance Assessment: Prospects and Barriers
Landon et al.
JAMA 2003;290:1183-1189.
ABSTRACT | FULL TEXT  

Commentary: Training in Pediatric Psychology: A Survey of Predoctoral Internship Programs
Kaslow and David
J Pediatr Psychol 2003;28:443-445.
FULL TEXT  

Toward Validation of an Assessment Tool Designed to Measure Medical Students' Integration of Scientific Knowledge and Clinical Communication Skills
Hugh Feeley et al.
Eval Health Prof 2003;26:222-233.
ABSTRACT  

Computer-based Testing in Family Practice Certification and Recertification
Hagen et al.
J Am Board Fam Med 2003;16:227-232.
ABSTRACT | FULL TEXT  

Multisource feedback: a method of assessing surgical practice
Violato et al.
BMJ 2003;326:546-548.
FULL TEXT  

Effectiveness of Medical Resident Education in Mechanical Ventilation
Cox et al.
Am. J. Respir. Crit. Care Med. 2003;167:32-38.
ABSTRACT | FULL TEXT  

Continuing Medical Education and the Physician as a Learner: Guide to the Evidence
Mazmanian and Davis
JAMA 2002;288:1057-1060.
FULL TEXT  

Competence Is a Habit
Leach
JAMA 2002;287:243-244.
FULL TEXT  

Tobacco Control Competencies for US Medical Students
Geller et al.
Am. J. Public Health 2005;95:950-955.
ABSTRACT | FULL TEXT