Theoretical Insights into Overparameterized Models in Multi-Task and Replay-Based Continual Learning

TL;DR

This paper provides a theoretical analysis of overparameterized models in multi-task and replay-based continual learning, revealing how system parameters influence generalization, transfer, and forgetting, with empirical validation on neural networks.

Contribution

It offers the first theoretical framework for understanding overparameterized models in multi-task and continual learning, connecting theory with practical deep learning applications.

Findings

Model size and task similarity affect generalization error.

Buffer size influences forgetting rate in continual learning.

Theoretical insights align with empirical results on neural networks.

Abstract

Multi-task learning (MTL) is a machine learning paradigm that aims to improve the generalization performance of a model on multiple related tasks by training it simultaneously on those tasks. Unlike MTL, where the model has instant access to the training data of all tasks, continual learning (CL) involves adapting to new sequentially arriving tasks over time without forgetting the previously acquired knowledge. Despite the wide practical adoption of CL and MTL and extensive literature on both areas, there remains a gap in the theoretical understanding of these methods when used with overparameterized models such as deep neural networks. This paper studies the overparameterized linear models as a proxy for more complex models. We develop theoretical results describing the effect of various system parameters on the model's performance in an MTL setup. Specifically, we study the impact of model size, dataset size, and task similarity on the generalization error and knowledge transfer. Additionally, we present theoretical results to characterize the performance of replay-based CL models. Our results reveal the impact of buffer size and model capacity on the forgetting rate in a CL setup and help shed light on some of the state-of-the-art CL methods. Finally, through extensive empirical evaluations, we demonstrate that our theoretical findings are also applicable to deep neural networks, offering valuable guidance for designing MTL and CL models in practice.