Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics

TL;DR

This study compares traditional and rubric-based graduate admissions in physics, finding that rubric-based methods fundamentally alter decision factors and could promote greater equity in admissions.

Contribution

It provides the first empirical analysis showing that rubric-based holistic review changes the admissions process compared to traditional methods.

Findings

Faculty relied on GRE scores and GPA before rubric implementation.

Models could predict admissions decisions before but not after rubric adoption.

Rubric-based review appears to fundamentally change the admissions process.

Abstract

Rubric-based admissions are claimed to help make the graduate admissions process more equitable, possibly helping to address the historical and ongoing inequities in the U.S. physics graduate school admissions process that have often excluded applicants from minoritized races, ethnicities, genders, and backgrounds. Yet, no studies have examined whether rubric-based admissions methods represent a fundamental change of the admissions process or simply represent a new tool that achieves the same outcome. To address that, we developed supervised machine learning models of graduate admissions data collected from our department over a seven-year period. During the first four years, our department used a traditional admissions process and switched to a rubric-based process for the following three years, allowing us to compare which parts of the applications were used to drive admissions decisions. We find that faculty focused on applicants' physics GRE scores and grade point averages when making admissions decisions before the implementation of the rubric. While we were able to develop a sufficiently good model whose results we could trust for the data before the implementation of the rubric, we were unable to do so for the data collected after the implementation of the rubric, despite multiple modifications to the algorithms and data such as implementing Tomek Links. Our inability to model the second data set despite being able to model the first combined with model comparison analyses suggests that rubric-based admissions does change the underlying process. These results suggest that rubric-based holistic review is a method that could make the graduate admissions process in physics more equitable.