Variability in Standardized Letters of Evaluation: A Multi‐Institutional Review of EM Residency Based Versus Non‐Residency Based Faculty Evaluation

TL;DR

This study compares how residency-based and non-residency-based faculty rate applicants in Emergency Medicine using standardized evaluation letters.

Contribution

The study reveals differences in score distributions between residency-based and non-residency-based faculty in Emergency Medicine evaluations.

Findings

Non-residency-based faculty assigned higher proportions of higher scores compared to residency-based faculty.

There was a positive relationship between evaluation scores and faculty estimates of anticipated guidance.

Most score items showed significant differences in proportions between the two faculty groups.

Abstract

The Standardized Letter of Evaluation (SLOE) in Emergency Medicine (EM) was developed by the Council of Residency Directors in Emergency Medicine (CORD) to have a uniform approach to providing training programs with information about applicants in the match process. Recent revisions distinguish what setting letter writers originate (residency based [RB] training program vs. non‐residency based [nRB] site).

The goal of this paper was to compare SLOE 2.0 scoring between letter writers from RB versus nRB settings.

This was a multi‐institutional cross‐sectional study. The study team from five residency programs collected data from SLOEs in their applicant pool from 2022 to 2023 match cycle. Each SLOE was reviewed for training location/SLOE type (RB vs. nRB) and numerical scores for sections A, B, and C (Anticipated Guidance [AG]). The data were not normally distributed, so were analyzed using descriptive and chi‐squared statistics. Data were examined using Spearman's Rho (⍴) to evaluate the relationship of Part A and B scores with faculty estimates of AG.

The study analyzed 3687 eSLOEs from 1772 applicants. The majority (N = 3526) were from RB faculty with only 161 from nRB faculty. The median scores were similar between groups, but the distribution of Part A and B scores was different between RB and nRB faculty. One exception was A4 (ability to perform common ED procedures) which had similar proportions of scores. There was a positive monotonic relationship between evaluation scores (Part A and B) and faculty estimates for AG, predicting up to 30% variability. Overall eSLOEs from nRB faculty had higher proportions of higher scores.

This study found a significant difference in proportions of scores assigned between RB faculty versus nRB faculty on most items including AG.