 |
|
Self-assessment in Lifelong Learning and Improving Performance in Practice
Physician Know Thyself
F. Daniel Duffy, MD;
Eric S. Holmboe, MD
JAMA. 2006;296:1137-1139.
A self-regulating profession holds its members accountable to the public it serves for the continuous development of the competencies they profess to hold. A central component of physician competence is professionalism, which requires lifelong learning that leads to improved performance in practice. A medical profession accomplishes accountability by providing its members periodic measurement of performance using reliable and valid instruments and judging performance against evidence-based standards, providing graduate and continuing medical education (CME) programs that advance members' knowledge and skills to meet these standards, and publicly certifying those who do so.
Successful completion of accredited graduate medical education and board certification is the bedrock evidence used for identifying initial physician competence; however, most of a physician's career is spent in practice using unstructured CME that relies on self-assessment to determine learning and improvement needs. Many state medical licensing boards and most certifying boards require evidence of periodic participation in self-selected CME programs to maintain a medical license and board certification. The systematic review of physician self-assessment by Davis and colleagues1 in this issue of JAMA provokes rethinking of whether it is wise to rely on unguided physician self-assessment as a cornerstone of continuous professional development.
Clinical education is rooted in experiential learning. Physicians learn from their patients. Educational theory and empirical research demonstrate that clinicians develop competence in their work by learning from their mistakes in performance.2-3 Advancing to expert levels of competence will not happen by reviewing failures in secret and making personal corrections; it needs guided feedback from other experts. The feedback needs to be based on an accurate appraisal of performance that identifies areas for expanding knowledge or improving methods of work.4 The medical profession's overemphasis on self-directed learning, self-assessment, and self-regulation by individual physicians may be one cause of ineffective learning over a career if physicians inaccurately assess their learning needs.5 Ineffective self-assessment can lead physicians to forfeit the necessary motivation to change their concepts, knowledge, or methods of care.
Davis et al1 demonstrate that there is a paucity of good studies to determine whether physicians are accurate in conducting self-assessment of predictive, summative, and concurrent performance compared with external indicators. Their analysis suggests a poor relationship between physician self-ratings of performance and the ratings provided by external raters. Even more worrisome is the finding that this inaccuracy may be worse for the least competent physicians who overestimate their competence. Such an error could lead to a failure to change ideas or practices and could sustain an unwarranted sense of competence.
The process and terminology of "self-assessment" is somewhat complex, and it is important to be clear about its usage and meaning.4 Davis et al1 focus on self-assessment as the ability of physicians to perform "self-rating" or use "self-audit" with a goal of generating summary judgments of their performance to determine their own learning needs and find resources to meet them. Their study thus excludes self-assessment programs that use questions to stimulate learning.
When US internists think of self-assessment, they may be more likely to consider the American College of Physicians' Medical Knowledge Self-Assessment Program, which is a type of self-assessment that Davis et al1 excluded. This take-home test is composed of single-best answer, multiple choice questions that are linked to a rationale for correct and incorrect answers and to a detailed, referenced syllabus. For nearly 40 years, the Medical Knowledge Self-Assessment Program has provided internists a means for measuring the currency of their clinical knowledge and judgment by answering questions derived from short-form cases.6 Many CME programs offer similar self-assessment programs. The reliability and validity of the self-assessment program as a cognitive measurement instrument can be determined by applying standard psychometric principles, although in most cases the purpose of these programs is to stimulate learning rather than to compare the individual's performance with others.
When assessment results are judged against an acceptable or passing performance level, the process becomes an "evaluation." Self-assessment as Davis et al use the term is more accurately defined as a self-evaluation because it requires that the physician develop a judgment about his or her grade of performance. The correctness of this rating was determined by its correlation with ratings on a measure of performance such as an objective structured clinical examination or by an external judge or panel of judges using the same assessment instrument.
Another meaning of self-assessment, as used in CME, could more accurately be called "guided self-audit," which refers to the activities physicians personally perform to assess their level of competence. Self-audit is an active process of looking systematically at the product of the physician's work (as in chart reviews) or clinical judgments (as in answering multiple choice questions) in contrast to the potentially more passive process of self-rating performance on a clinical examination or solution of a clinical problem. This latter process involves guesswork rather than the analysis of data.
Uncovering a gap in knowledge or in clinical performance motivates self-directed professionals to take action to close it. When the gap is discovered through self-assessment or self-audit, it seems to have more salience than one exposed by someone else. The American Board of Internal Medicine's initial experience with physicians completing a practice improvement module, which includes self-audit of medical records to calculate quality of care measures, lends additional support to the conclusions of Davis et al.1 When physicians received their results in structured feedback, they often experienced an aha moment. They saw and felt the gap in their performance compared with their impression that they were doing much better. Because physicians personally collected the data for measurement, they saw the gaps in performance for individual patients and the structured feedback for the sample was powerful and credible.7
When physicians receive credible feedback about their performance, they experience knowledge-performance discordance. This occurs in situations in which knowledge about the right thing to do is high and persons believe they are doing it, but when receiving feedback about actual performance they have a more accurate assessment of reality. Additionally, individuals experience emotional discomfort with this discordance, and if handled supportively, the discomfort provides the necessary motivation to align performance with knowledge about the guidelines for care. One problem with the emotional pain of the knowledge-performance discordance gap is the activation of psychological defense mechanisms of denial, rationalization, excuse making, and blaming that displaces the facts of a performance gap elsewhere.
One reason clinicians may not be good at recognizing deficiencies is that they often confuse confidence with competence. Self-assessment of confidence in a particular performance is an area of considerable concern. Confidence is a quality of self-efficacy that tends to correlate in empirical studies with persistence in the face of obstacles and higher achievement.8 Ratings of confidence can provide a baseline for guiding the structure of feedback provided to novice learners. However, the discordance between the overconfidence of the novice and the underconfidence of those achieving competence makes the unguided use of confidence measures particularly risky in medicine. The medical culture of "see one, do one, teach one" not only overemphasizes confidence to the detriment of acquiring true competence, but likely leads to the erroneous conclusion that actual performance data are not needed.9
The studies reported in the review by Davis et al1 demonstrate that physicians can reliably self-administer assessments of competence. To achieve the goals of public accountability, however, these assessments must be carefully structured and externally audited to ensure their veracity. Furthermore, when it comes to self-evaluation, applying personally determined standards for acceptable performance is risky and undesirable. This is particularly so in light of the knowledge that the least competent physicians, as judged by reliable external standards, tend to overrate their abilities. Physicians are not very good at knowing what they do not know or estimating how well they do know. This reinforces the observation by Eva and Regehr4 that an accurate judgment of performance cannot be made without standard measures based on credible data. Self-evaluation in the absence of credible data is unlikely to be of much value. The role of the self-audit, using accurate performance measurement and receiving expert-guided feedback about the measurement results, becomes an extremely important area for future work in continuous professional development.
Despite current limitations, self-assessment remains an essential tool for enabling physicians to discover the motivational discomfort of a performance gap, which may lead to changing concepts and mental models or changing work-flow processes. However, physicians must acknowledge that credible and reliable regulation of this assessment will be needed to ensure public accountability for fulfilling the social contract.10
As Davis et al1 recommend, the medical profession should shift CME and maintenance of certification and licensure to include processes of testing and educational methods that provide iterative feedback to supportively guide physicians in learning new concepts, as well as in changing work processes that lead to better patient care. This approach to continuous professional development combines the motivating energy provided by uncovering gaps in performance with CME, recertification, and relicensure. Physicians cannot be expected to do this in isolation; they require educational service from specialty societies and academic centers, and need the credible measurement tools and standards from certifying and licensing boards and organizations.
Guided self-assessment should be incorporated at the earliest stages of medical training as an essential professional skill.11 Self-audit should be strategically used in continuing professional development. Certifying boards are currently developing approaches to self-audit that should provide needed information on best practices and outcomes of patient care. These tools are being used for learning and improvement.12-13 As such tools become more robust, the medical profession must be prepared to deal effectively with physicians demonstrating low levels of competence and who are unable and unwilling to participate in meaningful self-assessment. The public expects nothing less.
Finally, there is an urgent need for more and better research into self-assessment, self-evaluation, and self-audit. Some important questions for further study include what happens when physicians choose the domain for self-assessment, whether this improves their accuracy, and whether repetitive self-assessment improves performance over time. Answers to these and the other questions posed by Davis et al1 will help to raise the standards for the entire profession.
AUTHOR INFORMATION
Corresponding Author: F. Daniel Duffy, MD, American Board of Internal Medicine, 510 Walnut St, Philadelphia, PA 19010 (dduffy{at}abim.org).
Financial Disclosures: Drs Duffy and Holmboe are employees of the American Board of Internal Medicine, which uses self-assessment and self-audit in its maintenance of certification process. No other financial disclosures were reported.
Disclaimer: The views expressed in this Editorial are the authors' own and do not reflect the official policy of the American Board of Internal Medicine.
Editorials represent the opinions of the authors and JAMA and not those of the American Medical Association.
Author Affiliations: American Board of Internal Medicine, Philadelphia, Pa.
REFERENCES
|
1. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296:1094-1102.
FREE FULL TEXT
2. Schon DA. The Reflective Practitioner. New York, NY: Basic Books; 1983.3. Dreyfus HL. On the Internet: Thinking in Action. New York, NY: Routeledge; 2002.4. Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med. 2005;80:S46-S54.
FULL TEXT
|
ISI
| PUBMED
5. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260-273.
FREE FULL TEXT
6. American College of Physicians. Medical knowledge self-assessment program. http://www.acp.online.org. Accessed July 31, 2006.7. Holmboe ES, Meehan TP, Lynn L, Doyle P, Sherwin T, Duffy FD. The ABIM Diabetes Practice Improvement Module: a new method for self assessment. J Continuing Educ Health Professions. 2006;26:109-119.
FULL TEXT
|
ISI
| PUBMED
8. Barnsley L, Lyon PM, Ralston SJ, et al. Clinical skills in junior medical officers: a comparison of self-reported confidence and observed competence. Med Educ. 2004;38:358-367.
FULL TEXT
|
ISI
| PUBMED
9. Debowski S, Wood RE, Bandura A. Impact of guided exploration and enactive exploration on self-regulatory mechanisms and information acquisition through electronic search. J Appl Psychol. 2001;86:1129-1141.
FULL TEXT
|
ISI
| PUBMED
10. American Board of Internal Medicine Foundation; American College of Physicians and European Federation of Internal Medicine. Medical professionalism in the millennium: a physician charter. Ann Intern Med. 2002;136:243-246.
FREE FULL TEXT
11. Holmboe ES, Prince L, Green ML. Teaching and improving quality of care in a residency clinic. Acad Med. 2005;80:571-577.
FULL TEXT
|
ISI
| PUBMED
12. Wasserman SI, Kimball HR, Duffy FD, Task Force on Recertification. Recertification in internal medicine: a program of continuous professional development. Ann Intern Med. 2000;133:202-208.
FREE FULL TEXT
13. Brennan TA, Horowitz RI, Duffy FD, Cassel CK, Goode LD, Lipner RS. The role of physician specialty board certification status in the quality movement. JAMA. 2004;292:1038-1043.
FREE FULL TEXT
RELATED ARTICLE
Accuracy of Physician Self-assessment Compared With Observed Measures of Competence: A Systematic Review
David A. Davis, Paul E. Mazmanian, Michael Fordis, R. Van Harrison, Kevin E. Thorpe, and Laure Perrier
JAMA. 2006;296(9):1094-1102.
ABSTRACT
| FULL TEXT
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
Dental Students' Self-Assessment of Preclinical Examinations
Curtis et al.
J Dent Educ 2008;72:265-277.
ABSTRACT
| FULL TEXT
Improving Asthma Care Through Recertification: A Cluster Randomized Trial
Simpkins et al.
Arch Intern Med 2007;167:2240-2248.
ABSTRACT
| FULL TEXT
|
