Authors

1. Sears, Kim RN, PhD
2. M, Christina Godfrey, RN, PhD
3. Luctkar, Marian Flude, RN, MScN (PhD candidate)
4. Ginsburg, Liane RN, PhD
5. Tregunno, Deborah RN, PhD
6. Ross-White, Amanda MLIS, AHIP

Abstract

Review question/objective: The objective of this systematic review is to compare the use of self-assessment instruments versus an objective structured clinical examination (OSCE) to measure the competence of healthcare learners and healthcare professionals. The specific question that will guide this review is: When measuring the competence of healthcare learners and healthcare professionals is the evaluation obtained by self-assessment instruments comparable to performance on an OSCE?

Background: Establishing the effectiveness of the health professional education process is complex and requires a multifaceted approach to assess the outcomes.1 Typically, outcomes are assessed in terms of the competence of the professional, level of confidence, performance and/or skills. Throughout the literature on this topic, these terms are used interchangeably, but there is overlap and some terms may encompass others. It is important to begin by providing descriptions/definitions of these terms.

Competence

In their paper that discusses the definition and assessment of professional competence, Epstein and Hundert define professional competence as: "the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection in daily practice for the benefit of the individual and community being served."2^(p226)

Confidence

Holland's concept analysis of professional confidence describes four components, namely affect (feelings associated with action), reflection (thoughtfully examine one's actions and intentions), higher cognitive functioning (which includes aspects such as learning and integration of concepts, decision making, attention, motivation and memory) and action.3^(p219)

Performance

The on-line Merriam-Webster dictionary defines performance as "the execution of an action or something accomplished - a deed or feat".4

Skill

Skill is defined as "proficiency, facility, or dexterity that is acquired or developed through training or experience."5

Self-efficacy

Self-efficacy is defined as "people's beliefs about their capabilities to produce designated levels of performance that exercise influence over events that affect their lives."6

Knowledge

Knowledge is defined as "(1) the fact or condition of knowing something with familiarity gained through experience or association; (2) acquaintance with or understanding of a science, art, or technique."7

Empathy

Empathy is defined as the "ability to understand and share the feelings of another."8

Looking at the above definitions of competence, confidence and performance it is clear that there is considerable overlap. The term competence was found to be the most inclusive. However, because there are nuances involved in each term, in this review the following concepts will be referred to: that of competence (including knowledge and performance) and confidence (including self-efficacy).

There are a variety of ways to measure health professionals' competence and self-assessment is a cost effective source of information. Self-assessment has been defined as: the evaluation or judgment of 'the worth' of one's performance and the identification of one's strengths and weaknesses with a view to improving one's learning outcomes.9^(p146) For example, self-reported patient safety competence may provide data about learners' insights into and likely safety of, their own practice,10 and about their perceived strengths or limitations.11,12 The value of using more objective methods to assess competence is unclear. Recent studies examining self versus expert assessment of technical and non-technical skills have produced mixed results. Surgeons seem to be able to accurately assess their own technical skills but not their non-technical skills;13 however, an earlier study of junior medical officers found no correlation between their self-assessments of confidence and their measured competencies on routine procedural skills.14

In a systematic review published in 2006, Davis and colleagues describe how accurately health professionals, primarily physicians, subjectively evaluate their own competence compared with external observations of their competence.15 They concluded that physicians have a limited ability to accurately self-assess and this may be particularly true among those rated as the least skilled and those who were also the most confident. These results were found to be consistent with other professions.16

The objective structured clinical examination (OSCE) is another method that has shown to be a useful means to assess the competence of a learner. Typically an OSCE consists of a specific scenario established by the examiners that requires the learner to demonstrate their proficiency in that area. The evaluator can control the environment and standardize the patient and in this manner use the OSCE to objectively assess competencies (i.e. knowledge, attitudes and behaviors). There is growing recognition that OSCEs are appropriate for evaluating the interpersonal skills associated with breaking bad news or cross-cultural interviewing.17 The use of the OSCE to assess physician communication skills is also becoming more common (e.g. Huntley, Salmon, Fisher, Fletcher, & Young;18 Iramaneerat, Myford, Yudkowsky, & Lowenstein;19 Ponton-Carss, Hutchison, & Violato;20 Van Nuland, Van den Noortgate, van der Vleuten, & Jo.21)

In the realm of patient safety there is a small but emerging body of literature encouraging the use of OSCEs to assess aspects of patient safety competence among medical trainees.22-28 In this area most OSCEs assess the technical aspects of patient safety or quality improvement competence,25,27-29 or clinical aspects of patient safety such as hand hygiene compliance and medication labeling.25 Few studies describe the use of OSCEs to assess socio-cultural aspects of patient safety,24,26 and those that do tend to focus on communicating or disclosing an error and are discipline-specific in nature.30,31

In nursing, a recent integrative review by Walsh and colleagues located 41 papers and identified major gaps regarding the psychometrics of nursing OSCEs.32 In concluding their review, the researchers highlighted the need for additional research on using the OSCE as an evaluative tool in nursing.

The OSCE is thought to be a more objective measure than self-assessment. However, while limited, examinations of the extent to which OSCE performance predicts outcomes on other performance metrics are somewhat equivocal. Some studies have failed to detect a significant positive relationship between OSCE performance and other forms of summative evaluations of health professionals and learners.23 A study by Tamblyn found that scores achieved in a patient-physician communication and clinical decision-making OSCE, that was part of a national licensing examination, predicted complaints to medical regulatory authorities up to 10 years later.33

In an environment where providing optimal student learning and quality patient care is a goal, there is a need to explore whether a link exists between self-assessment scores and OSCEs in light of providing the best learning for the most affordable means. It has been noted that some studies comparing self and external assessments of competence (such as the OSCE) have had several methodological problems. Davis and colleagues report that fewer than half of the studies they included in their systematic review (1) used pretested or validated OSCEs or standardized patients or assessment instruments, or (2) described objective criteria for performance assessment.15 Others have noted there is insufficient methodological detail in most published research involving standardized patients (SP), in particular details pertaining to SP characteristics and their training.

An examination of the Cochrane Library of Systematic Reviews, the Joanna Briggs Institute Database of Systematic Reviews and Implementation Reports and the PROSPERO database indicates that no systematic reviews have been completed (or proposed) on this topic since the Davis review in 2006. Building on the Davis review, this systematic review will explore research that examines the relationship between self-assessed competence and objective assessments of competence using the OSCE. The proposed synthesis is part of a broader program of research which builds on recommendations from numerous international bodies regarding the need to restructure health professional education to ensure it equips learners with the knowledge, skills and attitudes they need to function safely.1,34-37 Notably, there is also recognition that what is evaluated drives what is taught and learnt.38,39 Accordingly, development of an OSCE for adoption by various health professional education programs may be crucial for truly integrating patient safety into health professional education. Just as written examinations and OSCEs assess different things,40,41 so do subjective and objective assessments; however, both are understood to yield important data.10