ORFit: One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares

TL;DR

This paper introduces ORFit, a one-pass learning algorithm that efficiently updates models with new data by orthogonal gradient fitting, combining ideas from recursive least squares and principal component analysis to minimize forgetting and improve efficiency.

Contribution

The paper proposes ORFit, a novel one-pass learning method that bridges orthogonal gradient descent and recursive least squares, with theoretical and practical advantages for overparameterized models.

Findings

ORFit achieves perfect fit for new data while minimally affecting past predictions.

The algorithm is computationally efficient, with linear complexity in the number of parameters.

ORFit's parameter estimates match multi-pass SGD in overparameterized linear models.

Abstract

While large machine learning models have shown remarkable performance in various domains, their training typically requires iterating for many passes over the training data. However, due to computational and memory constraints and potential privacy concerns, storing and accessing all the data is impractical in many real-world scenarios where the data arrives in a stream. In this paper, we investigate the problem of one-pass learning, in which a model is trained on sequentially arriving data without retraining on previous datapoints. Motivated by the demonstrated effectiveness of overparameterized models and the phenomenon of benign overfitting, we propose Orthogonal Recursive Fitting (ORFit), an algorithm for one-pass learning which seeks to perfectly fit each new datapoint while minimally altering the predictions on previous datapoints. ORFit updates the parameters in a direction orthogonal to past gradients, similar to orthogonal gradient descent (OGD) in continual learning. We show that, interestingly, ORFit's update leads to an operation similar to the recursive least-squares (RLS) algorithm in adaptive filtering but with significantly improved memory and computational efficiency, i.e., linear, instead of quadratic, in the number of parameters. To further reduce memory usage, we leverage the structure of the streaming data via an incremental principal component analysis (IPCA). We show that using the principal components is minimax optimal, i.e., it minimizes the worst-case forgetting of previous predictions for unknown future updates. Further, we prove that, for overparameterized linear models, the parameter vector obtained by ORFit matches what the standard multi-pass stochastic gradient descent (SGD) would converge to. Finally, we extend our results to the nonlinear setting for highly overparameterized models, relevant for deep learning.