Ridge Boosting is Both Robust and Efficient

TL;DR

Ridge boosting, a simple one-step kernel ridge regression method, achieves both statistical efficiency and robustness to distributional shifts within an RKHS, enabling practical multi-target estimation with a single model.

Contribution

The paper introduces ridge boosting as a novel estimator that combines efficiency and robustness, bridging distinct research areas and simplifying multi-target estimation.

Findings

Achieves low bias across distribution shifts within an RKHS

Maintains variance at the semiparametric efficiency bound

Enables training a single model for multiple targets

Abstract

Estimators in statistics and machine learning must typically trade off between efficiency, having low variance for a fixed target, and distributional robustness, such as multiaccuracy, or having low bias over a range of possible targets. In this paper, we consider a simple estimator, ridge boosting: starting with any initial predictor, perform a single boosting step with (kernel) ridge regression. Surprisingly, we show that ridge boosting simultaneously achieves both efficiency and distributional robustness: for target distribution shifts that lie within an RKHS unit ball, this estimator maintains low bias across all such shifts and has variance at the semiparametric efficiency bound for each target. In addition to bridging otherwise distinct research areas, this result has immediate practical value. Since ridge boosting uses only data from the source distribution, researchers can train a single model to obtain both robust and efficient estimates for multiple target estimands at the same time, eliminating the need to fit separate semiparametric efficient estimators for each target. We assess this approach through simulations and an application estimating the age profile of retirement income.