Do GRE scores help predict getting a physics Ph.D.? A comment on a paper by Miller et al

TL;DR

This paper critiques a recent study claiming GRE scores do not predict physics Ph.D. completion, highlighting numerous statistical errors and methodological flaws that undermine its conclusions.

Contribution

It provides a detailed critique of the statistical methods used in Miller et al.'s study, emphasizing the importance of proper analysis techniques.

Findings

Identifies multiple statistical errors in the critique paper.

Highlights the need for correct data analysis in educational research.

Emphasizes the importance of proper null hypothesis testing.

Abstract

A recent paper in Sci. Adv. by Miller et al. concludes that GREs do not help predict whether physics grad students will get Ph.D.s. The paper makes numerous elementary statistics errors, including introduction of unnecessary collider-like stratification bias, variance inflation by collinearity and range restriction, omission of needed data (some subsequently provided), a peculiar choice of null hypothesis on subgroups, blurring the distinction between failure to reject a null and accepting a null, and an extraordinary procedure for radically inflating confidence intervals in a figure. Release of results of simpler models, e.g. without unnecessary stratification, would fix some key problems. The paper exhibits exactly the sort of research techniques which we should be teaching students to avoid.