Optimal Rates in Continual Linear Regression via Increasing Regularization

TL;DR

This paper demonstrates that increasing regularization in continual linear regression can achieve near-optimal convergence rates, closing the gap between known lower bounds and previous upper bounds, and provides practical strategies for optimal regularization schedules.

Contribution

The paper proves that increasing regularization schemes can attain optimal convergence rates in continual linear regression, improving upon prior bounds and guiding practical regularization schedules.

Findings

Regularization reduces the convergence rate gap in continual linear regression.

A fixed regularization strength achieves near-optimal $O(rac{ ext{log} k}{k})$ rate.

An increasing regularization schedule can attain the optimal $O(1/k)$ rate.

Abstract

We study realizable continual linear regression under random task orderings, a common setting for developing continual learning theory. In this setup, the worst-case expected loss after $k$ learning iterations admits a lower bound of $\Omega(1/k)$. However, prior work using an unregularized scheme has only established an upper bound of $O(1/k^{1/4})$, leaving a significant gap. Our paper proves that this gap can be narrowed, or even closed, using two frequently used regularization schemes: (1) explicit isotropic $\ell_2$ regularization, and (2) implicit regularization via finite step budgets. We show that these approaches, which are used in practice to mitigate forgetting, reduce to stochastic gradient descent (SGD) on carefully defined surrogate losses. Through this lens, we identify a fixed regularization strength that yields a near-optimal rate of $O(\log k / k)$. Moreover, formalizing and analyzing a generalized variant of SGD for time-varying functions, we derive an increasing regularization strength schedule that provably achieves an optimal rate of $O(1/k)$. This suggests that schedules that increase the regularization coefficient or decrease the number of steps per task are beneficial, at least in the worst case.