Objective scoring of application forms in obstetrics and gynaecology residency selection: A retrospective cohort study on the optimal number of committee members

TL;DR

This study finds that six assessors are enough to reliably score residency applications in obstetrics and gynaecology without sacrificing fairness.

Contribution

The study identifies the minimum number of assessors needed for reliable and efficient residency selection.

Findings

Six assessors provide reliable scores with correlations exceeding 0.9 compared to the grand average.

Adding more than six assessors does not significantly improve ranking consistency.

Using six assessors reduces workload while maintaining evaluation quality.

Abstract

The selection of residents for medical specialty programmes is a critical yet resource-intensive process. Although structured evaluation tools, such as standardized application forms, enhance objectivity and reliability, they often require all committee members to assess every candidate, resulting in inefficiencies. This study aimed to determine the optimal number of assessors needed to reliably score application forms of doctors to become resident in obstetrics & gynaecology without compromising selection outcomes.

This retrospective cohort study analysed data from three residency selection cycles (of the years 2022–2024) in the Northwest region of the Netherlands. Application forms were scored anonymously each year by 15–18 committee members, referred to as assessors, using a structured scoring system. Scores were analysed to identify the point at which adding more assessors no longer significantly impacted candidate rankings. Statistical measures included paired t-tests, correlations, and Cronbach’s alpha, and intraclass correlation coefficients to assess internal consistency and reliability.

The analysis showed that six assessors are sufficient to reliably assess candidates. Correlations between average scores from six assessors and the grand average consistently exceeded 0.9 across all cohorts, and Cronbach’s alpha stabilized above 0.85. Significant differences in rankings were observed when increasing assessors from two to six but diminished beyond six. Bland-Altman plots confirmed agreement between scores from six assessors and the overall committee evaluation.

A structured evaluation process (i.e., using standardized application forms) requiring six assessors per candidate ensures reliable and consistent outcomes while reducing workload. Implementing this approach can enhance efficiency without compromising fairness or objectivity in selection for obstetrics & gynaecology residents. Future research should investigate the applicability of this model to other medical residency programmes internationally, and its impact on long-term performance.

Figures