Administration of the American Board of Anesthesiology’s virtual APPLIED Examination: successes, challenges, and lessons learned

TL;DR

The American Board of Anesthesiology successfully administered a large-scale virtual exam during the pandemic, facing technical and security challenges but proving the feasibility of remote certification assessments.

Contribution

Demonstrates the practicality of virtual high-stakes exams in anesthesiology certification during the pandemic, highlighting both successes and limitations.

Findings

Over 80% of candidates completed the virtual exam without major issues, with only 2.5% of SOE and 1.5% of OSCE candidates needing rescheduling due to technical problems.

Approximately 40% of examiners found virtual evaluation more difficult than in-person, and 56% of SOE examiners considered the virtual format less secure.

The virtual exam format was eventually replaced by in-person exams in 2022 due to concerns about standardization and security.

Abstract

In response to the COVID-19 pandemic, the American Board of Anesthesiology transitioned from in-person to virtual administration of its APPLIED Examination, assessing more than 3000 candidates for certification purposes remotely in 2021. Four hundred examiners were involved in delivering and scoring Standardized Oral Examinations (SOEs) and Objective Structured Clinical Examinations (OSCEs). More than 80% of candidates started their exams on time and stayed connected throughout the exam without any problems. Only 74 (2.5%) SOE and 45 (1.5%) OSCE candidates required rescheduling due to technical difficulties. Of those who experienced “significant issues”, concerns with OSCE technical stations (interpretation of monitors and interpretation of echocardiograms) were reported most frequently (6% of candidates). In contrast, 23% of examiners “sometimes” lost connectivity during their multiple exam sessions, on a continuum from minor inconvenience to inability to continue. 84% of SOE candidates and 89% of OSCE candidates described “smooth” interactions with examiners and standardized patients/standardized clinicians, respectively. However, only 71% of SOE candidates and 75% of OSCE candidates considered themselves to be able to demonstrate their knowledge and skills without obstacles. When compared with their in-person experiences, approximately 40% of SOE examiners considered virtual evaluation to be more difficult than in-person evaluation and believed the remote format negatively affected their development as an examiner. The virtual format was considered to be less secure by 56% and 40% of SOE and OSCE examiners, respectively. The retirement of exam materials used virtually due to concern for compromise had implications for subsequent exam development. The return to in-person exams in 2022 was prompted by multiple factors, especially concerns regarding standardization and security. The technology is not yet perfect, especially for testing in-person communication skills and displaying dynamic exam materials. Nevertheless, the American Board of Anesthesiology’s experience demonstrated the feasibility of conducting large-scale, high-stakes oral and performance exams in a virtual format and highlighted the adaptability and dedication of candidates, examiners, and administering board staff.

The online version contains supplementary material available at 10.1186/s12909-024-05694-7.

Figures