Are Greedy Task Orderings Better Than Random in Continual Linear Regression?

TL;DR

This paper investigates whether greedy task orderings outperform random orderings in continual linear regression, providing theoretical bounds and empirical evidence that greedy orderings can converge faster but may also fail catastrophically under certain conditions.

Contribution

The paper formalizes greedy task orderings using Kaczmarz methods, compares their convergence properties to random orderings, and highlights conditions where greedy methods excel or fail.

Findings

Greedy orderings converge faster than random ones in empirical experiments.

In high-rank settings, loss bounds for greedy and random orderings are similar.

Single-pass greedy orderings can fail catastrophically, while repeated ones have better convergence rates.

Abstract

We analyze task orderings in continual learning for linear regression, assuming joint realizability of training data. We focus on orderings that greedily maximize dissimilarity between consecutive tasks, a concept briefly explored in prior work but still surrounded by open questions. Using tools from the Kaczmarz method literature, we formalize such orderings and develop geometric and algebraic intuitions around them. Empirically, we demonstrate that greedy orderings converge faster than random ones in terms of the average loss across tasks, both for linear regression with random data and for linear probing on CIFAR-100 classification tasks. Analytically, in a high-rank regression setting, we prove a loss bound for greedy orderings analogous to that of random ones. However, under general rank, we establish a repetition-dependent separation. Specifically, while prior work showed that for random orderings, with or without replacement, the average loss after $k$ iterations is bounded by $\mathcal{O}(1/\sqrt{k})$, we prove that single-pass greedy orderings may fail catastrophically, whereas those allowing repetition converge at rate $\mathcal{O}(1/\sqrt[3]{k})$. Overall, we reveal nuances within and between greedy and random orderings.