Utilising Random Effects Models to Analyse Multiple Mini-Interviews for Prospective Medical Students – From Theory to Practice

TL;DR

This paper introduces a statistical method and app to improve fairness and consistency in medical school interviews using random effects models.

Contribution

A user-friendly R Shiny app and cumulative probit mixed model for analyzing MMI data, accessible to non-statisticians.

Findings

Applicant ability contributed 22.94% of the total variance in MMI scores.

Interviewers contributed 10.79% of the variance, while station difficulty had a minor impact.

The app helps admissions staff provide feedback and identify effective interview stations.

Abstract

Multiple-mini-interviews (MMIs) are the most commonly used non-academic assessment tool for British medical school admissions processes. Potential inconsistencies can arise from running MMIs, such as differing marking standards among interviewers and stations with varying levels of difficulty.

With the aim of analysing MMI data, the cumulative probit mixed model was deployed which accounts for latent sources of variation inherent to MMI scores – both external factors, such as interviewer behaviour and station complexity, as well as the factor of interest – candidates’ true performance at interview. With the secondary aim of making this methodology more accessible to non-statistical experts, we developed a user-friendly application using R Shiny. The app was created to standardise MMI scores and generate feedback for interviewers without requiring prior knowledge of programming.

MMI data from Lancaster Medical School were analysed for the academic year 2022–2023. Applicant ability (n = 352) contributed to 22.94% of the total variance in MMI scores. Notably, interviewers (n = 83) contributed a smaller proportion of variance (10.79%). Station difficulty (n = 9) had a minor impact on variance (2.23%), with inter-station reliability being acceptable based on the Cronbach's alpha value (α = 0.7072).

The current method provides a statistically robust approach towards the analysis of MMI scores. In addition, its companion application can be used by admissions staff to communicate feedback to interviewers on their scoring patterns and identify stations that are better at discriminating applicants’ ability. The illustrated approach can be adapted for use by other medical institutions that use MMI scores to rank candidates during their admissions processes. Future research could investigate how to extend the proposed modelling approach to address applicants’ background factors, such as socio-economic status, for more comprehensive analysis and more equitable offer allocation.

Figures