Range restriction, admissions criteria, and correlation studies of standardized tests

TL;DR

This paper uses simulations to explore how range restriction and admissions criteria influence the observed correlations between standardized test scores and academic success, highlighting the importance of selection methods.

Contribution

It demonstrates how admissions criteria and range restriction affect the predictive validity of standardized tests and offers insights for interpreting correlation studies in admissions research.

Findings

Successful scientists with low GRE scores are consistent with multiple-variable performance models.

Observed correlations depend heavily on selection methods and weighting in admissions.

Range restriction effects vary with how student characteristics are weighted.

Abstract

Recent and influential critiques of standardized testing have noted the existence of non-trivial numbers of successful scientists who received low scores on the GRE. Here we use computer simulations to show that the prevalence of such examples is consistent with the reasonable hypothesis that academic performance depends on multiple variables. We examine the effects of admissions criteria on the observed predictive power of different variables, and show that observed correlations between student performance and student characteristics depend as much on the method of selecting students as on causal relationships detectable in the wider applicant pool. This is an example of the well-known statistical phenomenon of range restriction, and we offer relevant caveats and recommendations for further studies of admissions tests. We also show that the magnitude of range restriction effects depend on how student characteristics are weighted in admissions decisions.