Volume 9 no 1
Virtual objective structured clinical examinations — a novel approach to teaching and evaluating leadership skills in medical students
Michael Aw, MD, Ahmed Shoeib, MD, Craig Campbell, MD, and Charles Su, MD
Back to Index
Virtual objective structured clinical examinations — a novel approach to teaching and evaluating
leadership skills in medical students
Michael Aw, MD, Ahmed Shoeib, MD, Craig Campbell, MD, and Charles Su, MD
The objective structured clinical examination (OSCE) is a simulation-based method of learning and assessment that allows for mistakes and feedback. Its uniformity, objectivity, and reproducibility are among its greatest strengths. Unfortunately, OSCEs rarely directly assess non-clinical skills or focus on leadership skills, such as conflict management. Leadership training is an underrepresented component of the medical school curriculum. Although OSCEs cannot evaluate leadership in its entirety, we have demonstrated the feasibility of using OSCEs to simulate realistic scenarios for students to apply and practise communication, collaboration and professional skills associated with leadership.
KEY WORDS: leadership, simulation, examination, OSCE, evaluation
Aw M, Shoeib A, Campbell C, Su C. Virtual objective structured clinical examinations — a novel approach to teaching and evaluating leadership skills in medical students. Can J Physician Leadersh 9(1):12-20
doi://10.37964/cr24766
12,3
Simulation-based learning has also been shown to enhance non-technical competencies, such as foundational teamwork and empathy training among medical students.4,5 Unfortunately, OSCEs rarely directly assess non-clinical skills or focus on leadership skills, such as conflict management. Currently, no standardized method evaluates leadership skills in medical trainees.6
At the University of Ottawa, a student-run leadership elective was developed for pre-clerkship students. This program is a low-cost, 20-hour longitudinal leadership training program that uses a mix of teaching methods, specifically didactic and small-group learning, multisource feedback, and simulation-based learning in the form of OSCEs. The program’s objectives focus on leadership theory acquisition, enhanced knowledge of self, giving and receiving feedback, conflict management, persuasion, and advocacy skills. Unlike traditional OSCEs, which emphasize clinical knowledge, history taking, and physical examination competencies, these scenarios focus on non-clinical skills associated with leadership on an individual level.
To our knowledge, this initiative to provide simulation-based leadership training to medical students is the first of its kind in Canada. Here, we report on our experience with the development and implementation of a minimal resource leadership OSCE using resident evaluators. This framework may be useful as a guideline for replication at other medical institutions.
Developing leadership OSCEs
Content — Eight unique OSCE stations were developed in collaboration with leadership training experts at our medical institution to provide scenarios that a medical trainee might reasonably expect to experience in their clinical learning environment. The content was created to align with CanMEDS roles associated with effective leadership at an individual level: communicator (key competencies 1.1–5.3), collaborator (1.1–3.2), professional (1.1, 1.3, and 4.3), and leader (1.2 and 3.1).7 The skills needed to succeed in the scenarios were derived from the entrustable professional activities (EPAs) of the Association of Faculties of Medicine of Canada, specifically EPA 7 (provide and receive handover in transitions of care), EPA 9 (communicate in difficult conversations), EPA 10 (participate in health quality improvement situations), and EPA 12 (educate patients and families on management, promotion and prevention).8 The aim was to create two thematically comparable four-station circuits that were similar in difficulty and scope of skill assessment. Ultimately, four major scenarios were developed: hostile conflict management, shared decision-making, managing expectations, and advocating for others (Appendix 1, p. 17).
Evaluation — The OSCEs were assessed using the modified Oxford Non-Technical Skills (NOTECHS) scale as a guide to provide student feedback.9 The NOTECHS tool was used because it had been validated to assess the nontechnical performance of residents. It encompasses five domains: leadership and managerial skills, teamwork and cooperation, problem-solving and decision-making, communication and interaction, and situation awareness and vigilance.
Implementation — The OSCEs were conducted virtually using the Zoom platform (Zoom Video Communications Inc., San José, Calif., USA) with standardized patients. Zoom allows for users to be placed in breakout rooms that mimic real-life stations. Participants completed a circuit of four eight-minute stations with two-minute breaks, during which they were moved to the next breakout room. Participant consent was obtained, and all OSCE encounters were recorded for subsequent feedback by independent evaluators. After each OSCE circuit, participants engaged in a one-hour debrief to share their experiences and approaches for each station with their peers.
Standardized patients (SPs) — SP volunteers were recruited via social media postings. To ensure standardization of tone, language, and behavioural responses among SPs for each unique station, we created basic role descriptions with guiding scripts, included mandatory hallmark phrases and prompting statements to guide participants toward the scenario’s climax/conclusion if necessary (Appendix 2, p. 19). During a mandatory two-hour training session, SPs were asked to act out their roles with one another numerous times so that they could further attempt to standardize the demeanor of the SP character.
Evaluators — Program directors distributed a recruitment letter via email to University of Ottawa residents. Ultimately, eight residents agreed to participate in this pilot program. They completed a two-hour training session, in which they were shown video recordings from the pilot and came to a consensus on the benchmark performance quality for each domain described in the NOTECHS tools. Performances on each OSCE scenario were scored by two independent reviewers.
Ethics — Appropriate applications were submitted to our institution’s ethics committee from which
we received exemption under Article 2.5 of Tri-Council Policy Statement 2.
OSCE reception by participants
Sixteen first-year students participated in two four-station OSCEs. We solicited feedback and quantified satisfaction using a five-point Likert scale questionnaire (Figure 1). Narrative comments were summarized using qualitative analysis (Tables 1A and 1B). Major ideas were chronicled in preliminary topic coding. After reviewing the topic codes, thematic coding was performed in which descriptors were grouped according to common themes determined by the primary investigators.
Students generally responded positively to their OSCE experience. All participants agreed that the content of the OSCEs was relevant to leadership training. Specifically, 93% of participants agreed or strongly agreed that the OSCEs allowed them to practise their negotiation and conflict management skills (Figure 1). Nearly all students (96%) agreed or strongly agreed that, overall, the OSCEs provided an enjoyable learning experience.
The most frequently reported positive aspects of the virtual OSCEs related to the realism and relevance of the situations (41%) and the opportunity to apply non-clinical skills and knowledge (38%) (Table 1A and B).
Several students commented on the challenge associated with real-time scenarios, which encouraged them to practise their leadership skills. “This was an amazing and helpful session! It placed us under difficult circumstances with a lot of pressure, especially when the actors were unwilling to compromise. It was an extremely rewarding experience in terms of building communication, negotiation, and leadership skills.”
Students reported that their experience debriefing and discussing the cases with their peers was beneficial to their learning and to acquiring different perspectives and strategies to manage difficult situations. “I really loved this session and thought it was incredibly helpful! My favourite part was just having the opportunity to debrief afterwards and see how different people approached the situations.”
Areas of improvement pertained predominately to program logistics. Specifically, some students would have preferred live feedback from evaluators (43%), and some felt the time allotted to individual scenarios may have been insufficient (27%). “Receiving more immediate feedback following scenarios would have allowed for a fresher approach per scenario.” Likewise, students requested additional time to be allotted to the debrief discussions (22%).
OSCE reception by evaluators
Narrative comments were solicited from resident evaluators regarding the appropriateness of using the NOTECHS tool and their comfort with evaluating non-clinical skills, namely problem-solving, communication, and situational awareness (Table 2). In general, feedback from the evaluators was positive. Most evaluators described the scenarios as realistic and appropriately complex (82%) and agreed that they facilitated variable approaches to managing the conflict associated with individual scenarios (47%). Almost all evaluators reported the NOTECHS tool as user friendly (94%). Specific comments praised the tool’s utility for assessing problem-solving, communication, and situational awareness. “It was easy to grade and most elements fit well. Leadership was a bit difficult to grade based on the description; however, communication, cooperation, situation awareness fit really well.”
Discussion
The OSCE scenarios were well received, with most student participants reporting a positive learning experience. Participants reported that the content was relevant and they enjoyed the opportunity to apply and practise their leadership skills. Overall, resident evaluators felt comfortable assessing non-clinical skills and reported that the NOTECHS tool was appropriate for non-clinical, leadership skills assessment and directed feedback. Most constructive feedback related to OSCE logistics, namely the absence of real-time feedback and the desire for additional post-OSCE group discussion.
From our understanding, this is the first non-clinical, leadership skills OSCE delivered to undergraduate Canadian medical students. Varkey et al.10 discerned that 85% of medical student respondents identified leadership, conflict management, communication and teamwork as necessary skills that should be taught in medical school. Although recognized as important professional competencies, students are often left learning these non-technical leadership skills informally as part of the hidden curriculum.11 Notably, Bharwani et al.12 report communication skills and conflict resolution as notable skills missing from medical school leadership programs. Simulation provides the opportunity for students to practise their acquired knowledge of leadership skills and apply them in high-fidelity scenarios, providing a process for enhanced leadership training among undergraduate and graduate medical students.13,14 Generally, medical students prefer simulation exercises and facilitated small-group discussions as an effective teaching strategy compared with didactic lectures.10 This highlights the potential value of incorporating simulation-based exercises, such as the leadership OSCEs described here, into formal medical curricula.
A common challenge associated with OSCEs is the cost of SPs and evaluators. A four-hour OSCE for 120 students can cost in excess of $100 000.15 The major limitations of leadership training programs include human resources, staff training, location availability, as well as time and evaluation costs.16 As a student-led initiative with a limited budget, we relied on volunteer SPs and resident evaluators. Feedback from these volunteers revealed no concerns regarding standardization, lack of professionalism, or realism among the SPs. To facilitate the involvement of resident evaluators, pre-recording of encounters enabled all evaluators to assess scores independently without compromising their clinical responsibilities. Finally, the virtual platform minimized cost and the need for expensive, logistically challenging in-person examination rooms. Although some students said that the OSCEs would benefit from in-person delivery, previous literature suggests that, regarding communication skill assessment, virtual OSCEs are comparable to in-person sessions.17 Our successful piloting of OSCEs using volunteers via a virtual format may represent a practical method for other student-run initiatives to incorporate simulation-based learning into their programs.
The OSCE comes with challenges in terms of assessment of non-clinical skills. The use of standardized checklists limits the ability to assess non-clinical skills, such as conflict management, decision-making, and effective communication. The NOTECHS tool has been validated as effective in assessing non-clinical skills among residents in the context of an operating theatre.9 We used this tool as a guide to direct evaluator feedback for participants, and it was well received by the resident evaluators. Although not formally assessed here, the NOTECHS tool should be considered for non-clinical skill evaluation among undergraduate medical students.
This innovation was conducted as a pilot project, and the feedback reported here should not be generalized to all medical trainees or institutions. The innovation was limited by a small sample size, thus restricting the ability to draw definite conclusions. It may be advantageous to compare virtual OSCEs with in-person OSCEs to determine an optimal strategy for leadership skills training and assessment. Moreover, further studies are warranted to compare the NOTECHS tool with other non-clinical assessment measures to clarify optimal assessment of non-clinical skills among undergraduate medical trainees. More formal assessments of the quality and appropriateness of resident evaluators for non-technical communication skills is warranted.
Conclusion
Leadership training is an underrepresented component of the medical school curriculum. Leadership is a multifaceted and complex competency that encompasses many broad skills and attributes. Currently, there is no unified definition of leadership. Although OSCEs cannot evaluate leadership in its entirety, we have demonstrated the feasibility of using OSCEs to simulate realistic scenarios for students to apply and practise communication, collaboration, and professional skills associated with leadership.
This innovation was well received by student participants who reported a positive learning experience and valued the opportunity to practise their leadership skills. Resident evaluators endorsed the NOTECHS tool for assessment that was practical and cost-effective. Further formal evaluation is required to draw definitive conclusions and determine ideal assessment strategies to maximize the leadership training experience for medical students.
Appendix 1: OSCE scenarios
Baseline OSCE
1. Hostile conflict management
In this scenario, an interprofessional conflict arises between a fourth-year medical student (the student) and a nurse (the SP) with regard to orders that have not been completed by the nurse. The nurse has apprehensions about new medical students resulting from past negative interactions. The nurse’s primary intent is patient safety and prefers to work with staff over medical students for that reason and, therefore, does not take medical student co-signed orders seriously. The medical student must find a way to collaborate with the nurse and navigate this conflict.
2. Shared decision-making
In this scenario, a conflict arises between a fourth-year medical student (the student) and the father (the SP) of a 14-year-old who has been recommended for surgery to address underlying acute appendicitis. Although this is a very low-risk, safe procedure, the father is hesitant to have it carried out on his son, because of his past negative experience when his wife died during a complicated surgical procedure. The medical student must try to understand the father’s apprehensions, while demonstrating empathy and helping in the decision-making process to let this child have this life-saving procedure.
3. Managing expectations
In this scenario, an uncomfortable situation arises between a third-year clerk (the student) and a patient (the SP). In passing, the clerk tells the patient that her rash was “probably just allergies,” which reassures the patient. It is later discovered that the cause of the rash is lupus. The patient feels betrayed and misled by the student’s previous statements and is angry. The student must realize the importance of voicing such “reassuring” statements and the importance of being transparent with patients. The student must find a way to calm the patient down while accepting accountability for their error of judgement.
4. Advocating for others
In this scenario, a senior resident (the student) notices that their junior resident (the SP) has been showing up late to work. She looks disheveled and her bedside manner has deteriorated over the last few weeks. Reports from other residents who have worked with her show that in the morning there have been concerns that her breath reeks of alcohol, yet she denies consuming any substances before coming to work. The senior resident must confront this junior resident, gather further information, and advocate for the well-being of their peer and patients.
Final OSCE
1. Hostile conflict management
In this scenario, a disheveled patient (the SP) is brought into the hospital after falling off of their bike. He/she has suffered a fracture to his/her right arm. The patient is extremely distraught that they are being cared for by a female nurse of Caribbean descent. The patient has demanded to speak with his/her physician (the student) to switch nurses. Before you enter the room, you see the nurse crying after being told, “go back to your country, you don’t belong here.” The physician must talk with this patient, address the situation, and resolve the conflict.
2. Shared decision-making
In this scenario, a conflict arises between a fourth year clerk (the student) and a young adult diagnosed with acute lymphocytic leukemia (the SP) with regard to the proposed treatment plan. The student has been asked to share the evidence-based treatment plan proposed to care for the patient. However, the patient does not wish to undergo the allopathic treatment and instead wishes to pursue homeopathic treatment options. The patient insists that “Western medicine” is not always ideal and wishes to undergo homeopathic treatment instead, even though it might be fatal. The student must find a way to collaborate with the patient and educate him/her on the consequences of pursuing allopathic treatment options alone.
3. Managing expectations
In this scenario, your patient (Mr. Ruth) has advanced dementia and is approaching end-of-life care decisions. The goals of treatment have been switched to palliation by his designated power of attorney (spouse). His family include an adult son, adult daughter, and wife who have come to visit him (three separate households) all the way from rural Quebec (10-hour drive). The hospital policy is that no more than two family members from two separate households are allowed to visit a patient because of the COVID-19 pandemic. The nurse notices that the family members are not wearing masks while in the room with Mr. Ruth, who shares the room with two other vulnerable patients. The senior resident must talk with Mr. Ruth’s son/daughter and explain that hospital policy requires them to wear a mask and find a way to help educate all parties and facilitate compliance with hospital policies.
4. Advocating for others
In this scenario, a senior resident (the student) is told by the nurse that one of the junior residents has an unpleasant body odour and that a number of patients have started to complain. She notes this is not the first time this has happened. You, too, have noticed that the junior resident’s hair looks unclean and they smell of BO at your weekly rounds’ meetings. Your peers have started gossiping and talking about the stench behind the aforementioned resident’s back. The senior resident must address this situation with the junior resident, gather further information and reach a resolution while maintaining a positive interpersonal relationship.
Appendix 2: Sample OSCE script
Case 1: Interprofessional conflict
With multiple health professionals involved in the care of a patient, the potential for interprofessional conflict increases. Poorer patient outcomes have been linked to the inability of physicians and health care workers to use effective conflict-management skills to navigate these interprofessional conflict situations. This scenario is aimed at simulating an inter-professional conflict that a medical student is likely to experience.
Goals and objectives (based on Association of Faculties of Medicine of Canada EPA 9, EPA 5 and communicator/professional CANMEDS roles):
- Applies conflict-resolution skills in the setting of an interprofessional conflict
- Explores ways to facilitate collaborative decision-making between health care workers
- Uses effective communication skills during interprofessional conflict situations.
- Listens actively to colleague’s concerns and plans accordingly to attempt to explore options that satisfy both parties
Case description: In this scenario, an interprofessional conflict arises between a fourth-year medical student (the student) and a nurse (the SP) with regard to orders that have not been completed by the nurse. The nurse has apprehensions about new medical students resulting from past negative interactions. The nurse’s primary intent is patient safety and prefers to work with staff over medical students for that reason and, therefore, does not take medical student co-signed orders seriously. The medical student must find a way to collaborate with the nurse and navigate this conflict.
Nurse role: You are an inpatient nurse who works at the surgical ward at your local hospital. The latest newly minted clerkship student is approaching you to follow up on some orders they entered into the health record that have not been completed.
You have apprehensions and general “distrust” of new medical students and do not take their orders seriously. You have been working at this hospital for almost 20 years and have familiarity with many of the staff and have seen many residents come through your ward. You like the current “system” and find that working with staff is much better than working with these “not real” doctors. Your reasoning stems from your negative experience working with other medical students throughout the years where multiple medical errors have occurred and you want to prioritize patient safety.
You will maintain a confrontational demeanour and will be stubborn toward the medical student’s follow up to their orders at first. Focus on the negatives and your experiences in working with medical students in the past, but you are open to the medical student’s exploration of your reasoning (prioritizing patient safety). The medical student must show an understanding of your perspective before you proceed toward an openness for compromise.
Some sample probing statements:
- “I find medical students delay my ability to provide care/make too many medical errors for my patients, so I prefer to wait until I hear what the ‘real doctor’ says.”
- “I usually work directly with Dr. Lee and Dr. Smith.”
- “I’ve been here 20 years and we have a system that works well on our ward. I prefer to work with staff that I am familiar with.”
Resident (student) role: You are a fourth-year clerk who recently started on the inpatient surgical unit. Things have been going well; however, you are finding that the orders you entered into the health record after discussions with your resident and attending physician are either completed late or not completed at all by a certain nurse. You find that this is a good time to approach the nurse at the nursing station and follow up on some of these uncompleted orders.
Student “succeeds” if they
- Acknowledge the nurse’s perspective.
- State their stance as a learner and outline the shared responsibility of the nurse/resident toward patient care.
- Make an effort to reach a solution.
References
1.Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ 1979;13(1):39-54. https://doi.org/10.1111/J.1365-2923.1979.TB00918.X
2.Evans LV, Crimmins AC, Bonz JW, Gusberg RJ, Tsyrulnik A, Dziura JD, et al. A comprehensive, simulation-based approach to teaching clinical skills: the medical students’ perspective. Yale J Biol Med 2014;87(4):575-81.
3.Laack TA, Newman JS, Goyal DG, Torsher LC. A 1-week simulated internship course helps prepare medical students for transition to residency. Simul Healthc 2010;5(3):127-32. https://doi.org/10.1097/SIH.0b013e3181cd0679
4.Banerjee A, Slagle JM, Mercaldo ND, Booker R, Miller A, France DJ, et al. A simulation-based curriculum to introduce key teamwork principles to entering medical students. BMC Med Educ 2016;16(1):295. https://doi.org/10.1186/s12909-016-0808-9
5.Wündrich M, Schwartz C, Feige B, Lemper D, Nissen C, Voderholzer U. Empathy training in medical students — a randomized controlled trial. Med Teach 2017;39(10):1096-8. https://doi.org/10.1080/0142159X.2017.1355451
6.Lyons O, Su’a B, Locke M, Hill A. A systematic review of leadership training for medical students. N Z Med J 2018;131(1468):75-84.
7.Frank JR, Danoff D. The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach 2007;29(7):642-7. https://doi.org/10.1080/01421590701746983
8.AFMC entrustable professional activities for the transition from medical school to residency. Ottawa: Association of Faculties of Medicine of Canada; 2016. Available:
9.Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care 2009;18(2):104-8. https://doi.org/10.1136/QSHC.2007.024760
10.Varkey P, Peloquin J, Reed D, Lindor K, Harris I. Leadership curriculum in undergraduate medical education: a study of student and faculty perspectives. Med Teach 2009;31(3):244-50. https://doi.org/10.1080/01421590802144278
11.Cooke M, Irby DM, Sullivan W, Ludmerer KM. American medical education 100 years after the Flexner report. N Engl J Med 2006;355(13):1339-44. https://doi.org/10.1056/NEJMra055445
12.Bharwani A, Kline T, Patterson M. Perceptions of effective leadership in a medical school context. Can Med Educ J 2019;10(3):e101-6. Available: https://tinyurl.com/3bdtbn6x
13.Hunziker S, Bühlmann C, Tschan F, Balestra G, Legeret C, Schumacher C, et al. Brief leadership instructions improve cardiopulmonary resuscitation in a high-fidelity simulation: a randomized controlled trial. Crit Care Med 2010;38(4):1086-91. https://doi.org/10.1097/CCM.0B013E3181CF7383
14.Bearman M, O’Brien R, Anthony A, Civil I, Flanagan B, Jolly B, et al. Learning surgical communication, leadership and teamwork through simulation. J Surg Educ 2012;69(2):201-7. https://doi.org/10.1016/J.JSURG.2011.07.014
15.Reznick RK, Smee S, Baumber JS, Cohen R, Rothman A, Blackmore D, et al. Guidelines for estimating the real cost of an objective structured clinical examination. Acad Med 1993;68(7):513-7. https://doi.org/10.1097/00001888-199307000-00001
16.Richard K, Noujaim M, Thorndyke LE, Fischer MA. Preparing medical students to be physician leaders: a leadership training program for students designed and led by students. MedEdPORTAL 2019;15:10863. https://doi.org/10.15766/MEP_2374-8265.10863
17.Levine S, Strulovitch C, Ansari A, Baron A, Fetzer M, Fosler L, et al. Comparison of virtual versus live OSCE (objective structured cinical examination) within an interdisciplinary palliative care training program (QI410). J Pain Symptom Manage 2022;63(5):890. https://doi.org/10.1016/J.JPAINSYMMAN.2022.02.097
Authors
Michael Aw, BHSc, MD (candidate), is a medical student at the University of Ottawa, Faculty of Medicine.
Ahmed Shoeib, BSc, MD (candidate), is a medical student at the University of Ottawa, Faculty of Medicine.
Craig Campbell, MD, is an associate professor in the Department of Medicine and curriculum director for undergraduate medical education, University of Ottawa.
Charles Su, MD, is a professor in the Department of Emergency Medicine and Department of Family Medicine, and interim vice dean of undergraduate medical education, University of Ottawa.
No potential conflicts of interest relevant to this article were reported.
Author contributions: Conceptualization: AS, MA. Data curation: AS, MA. Methodology: AS, MA. Formal analysis: AS, MA. Supervision: CC, CS. Writing, original draft: AS, MA. Writing, review and editing: AS, MA, CC, CS. All authors approved the final article.
Correspondence to: