Predictive validities: figures of merit or veils of deception?

TL;DR

This paper critiques the validity estimates of the GRE, revealing methodological flaws and arguing for a utility-based approach that considers misclassification rates, which show the GRE may discriminate and produce errors worse than random decisions.

Contribution

It introduces a utility-focused method for evaluating tests, challenging traditional correlation-based validity estimates and highlighting biases in the GRE.

Findings

GRE validity estimates are inflated due to flawed methodology.

Conventional validity measures are incomplete without considering base-rates and quotas.

GRE may discriminate against minorities and lead to more errors than random selection.

Abstract

The ETS has recently released new estimates of validities of the GRE for predicting cumulative graduate GPA. They average in the middle thirties - twice as high as those previously reported by a number of independent investigators. It is shown in the first part of this paper that this unexpected finding can be traced to a flawed methodology that tends to inflate multiple correlation estimates, especially those of populations values near zero. Secondly, the issue of upward corrections of validity estimates for restriction of range is taken up. It is shown that they depend on assumptions that are rarely met by the data. Finally, it is argued more generally that conventional test theory, which is couched in terms of correlations and variances, is not only unnecessarily abstract but, more importantly, incomplete, since the practical utility of a test does not only depend on its validity, but also on base-rates and admission quotas. A more direct and conclusive method for gauging the utility of a test involves misclassification rates, and entirely dispenses with questionable assumptions and post-hoc "corrections". On applying this approach to the GRE, it emerges (1) that the GRE discriminates against ethnic and economic minorities, and (2) that it often produces more erroneous decisions than a purely random admissions policy would.