Highly Adaptive Ridge

TL;DR

The paper introduces Highly Adaptive Ridge (HAR), a nonparametric regression method with a dimension-free convergence rate, suitable for tabular data, and demonstrates its superior empirical performance over existing algorithms.

Contribution

HAR is a novel data-adaptive kernel ridge regression method that achieves optimal convergence rates for a broad class of functions.

Findings

HAR achieves a $n^{-1/3}$ convergence rate.

Empirical results show HAR outperforms state-of-the-art algorithms on small datasets.

Simulation confirms the theoretical convergence properties.

Abstract

In this paper we propose the Highly Adaptive Ridge (HAR): a regression method that achieves a $n^{-1/3}$ dimension-free L2 convergence rate in the class of right-continuous functions with square-integrable sectional derivatives. This is a large nonparametric function class that is particularly appropriate for tabular data. HAR is exactly kernel ridge regression with a specific data-adaptive kernel based on a saturated zero-order tensor-product spline basis expansion. We use simulation and real data to confirm our theory. We demonstrate empirical performance better than state-of-the-art algorithms for small datasets in particular.