Limitations of P-Values and $R^2$ for Stepwise Regression Building: A Fairness Demonstration in Health Policy Risk Adjustment

TL;DR

This paper critiques the reliance on p-values and R^2 in high-dimensional stepwise regression, highlighting potential fairness issues in health policy risk adjustment.

Contribution

It demonstrates the limitations of traditional fit measures in high-dimensional settings and emphasizes the importance of group fairness in health policy applications.

Findings

P-values and R^2 can mislead in high-dimensional stepwise regression.

Traditional fit measures may overlook fairness concerns in health policy.

Group-level fairness metrics reveal biases not captured by individual fit measures.

Abstract

Stepwise regression building procedures are commonly used applied statistical tools, despite their well-known drawbacks. While many of their limitations have been widely discussed in the literature, other aspects of the use of individual statistical fit measures, especially in high-dimensional stepwise regression settings, have not. Giving primacy to individual fit, as is done with p-values and $R^2$, when group fit may be the larger concern, can lead to misguided decision making. One of the most consequential uses of stepwise regression is in health care, where these tools allocate hundreds of billions of dollars to health plans enrolling individuals with different predicted health care costs. The main goal of this "risk adjustment" system is to convey incentives to health plans such that they provide health care services fairly, a component of which is not to discriminate in access or care for persons or groups likely to be expensive. We address some specific limitations of p-values and $R^2$ for high-dimensional stepwise regression in this policy problem through an illustrated example by additionally considering a group-level fairness metric.