Calibrated Principal Component Regression

TL;DR

This paper introduces Calibrated Principal Component Regression (CPCR), a novel method that reduces bias and improves prediction in high-dimensional generalized linear models by calibrating PCR with a Tikhonov step.

Contribution

The paper proposes CPCR, which enhances PCR by calibrating in the original feature space, controlling bias, and providing theoretical risk analysis in overparameterized regimes.

Findings

CPCR outperforms standard PCR in low-variance directions.

Empirical results show CPCR consistently improves prediction in overparameterized problems.

Theoretical analysis confirms CPCR's risk is lower than PCR's in certain regimes.

Abstract

We propose a new method for statistical inference in generalized linear models. In the overparameterized regime, Principal Component Regression (PCR) reduces variance by projecting high-dimensional data to a low-dimensional principal subspace before fitting. However, PCR incurs truncation bias whenever the true regression vector has mass outside the retained principal components (PC). To mitigate the bias, we propose Calibrated Principal Component Regression (CPCR), which first learns a low-variance prior in the PC subspace and then calibrates the model in the original feature space via a centered Tikhonov step. CPCR leverages cross-fitting and controls the truncation bias by softening PCR's hard cutoff. Theoretically, we calculate the out-of-sample risk in the random matrix regime, which shows that CPCR outperforms standard PCR when the regression signal has non-negligible components in low-variance directions. Empirically, CPCR consistently improves prediction across multiple overparameterized problems. The results highlight CPCR's stability and flexibility in modern overparameterized settings.