Highly Adaptive Principal Component Regression

TL;DR

This paper introduces PCHAL and PCHAR, new principal component-based estimators that improve computational efficiency of HAL and HAR while maintaining strong empirical performance.

Contribution

The paper develops principal component variants of HAL and HAR, enabling faster computation and comparable accuracy, along with a spectral regularization method via early stopping.

Findings

PCHAL and PCHAR achieve substantial computational gains.

They maintain empirical performance similar to HAL and HAR.

HAL kernel can be identical to Brownian motion covariance under certain conditions.

Abstract

The Highly Adaptive Lasso (HAL) is a nonparametric regression method that achieves almost dimension-free convergence rates under minimal smoothness assumptions, but its implementation can be computationally prohibitive in high dimensions due to the large design matrix it requires. The Highly Adaptive Ridge (HAR) has been proposed as a related ridge-regularized analogue. Building on both procedures, we introduce the Principal Component Highly Adaptive Lasso (PCHAL) and Principal Component Highly Adaptive Ridge (PCHAR). These estimators use an outcome-blind principal-component reduction of the HAL basis, offering substantial computational gains over HAL while achieving empirical performance comparable to HAL and HAR. We also describe an early-stopped gradient descent variant, which provides a convenient form of smooth spectral regularization without explicitly selecting a hard principal-component cutoff. Finally, we uncover that under special circumstances, the HAL kernel is identical to the covariance function of Brownian motion.