Adaptive Benign Overfitting (ABO): Overparameterized RLS for Online Learning in Non-stationary Time-series

TL;DR

This paper introduces Adaptive Benign Overfitting (ABO), a stable online learning method for non-stationary time-series that combines RLS with orthogonal-triangular updates, achieving high accuracy and efficiency in overparameterized models.

Contribution

The work extends RLS to the benign overfitting regime using a numerically stable QR-based approach with Fourier features and forgetting factors for non-stationary data.

Findings

Maintains bounded residuals and stable condition numbers in synthetic experiments.

Achieves high accuracy comparable to kernel methods in real-world forecasting tasks.

Improves computational speed by 20-40% over baseline methods.

Abstract

Overparameterized models have recently challenged conventional learning theory by exhibiting improved generalization beyond the interpolation limit, a phenomenon known as benign overfitting. This work introduces Adaptive Benign Overfitting (ABO), extending the recursive least-squares (RLS) framework to this regime through a numerically stable formulation based on orthogonal-triangular updates. A QR-based exponentially weighted RLS (QR-EWRLS) algorithm is introduced, combining random Fourier feature mappings with forgetting-factor regularization to enable online adaptation under non-stationary conditions. The orthogonal decomposition prevents the numerical divergence associated with covariance-form RLS while retaining adaptability to evolving data distributions. Experiments on nonlinear synthetic time series confirm that the proposed approach maintains bounded residuals and stable condition numbers while reproducing the double-descent behavior characteristic of overparameterized models. Applications to forecasting foreign exchange and electricity demand show that ABO is highly accurate (comparable to baseline kernel methods) while achieving speed improvements of between 20 and 40 percent. The results provide a unified view linking adaptive filtering, kernel approximation, and benign overfitting within a stable online learning framework.