Continual Learning with Distributed Optimization: Does CoCoA Forget?

TL;DR

This paper investigates whether the distributed optimization algorithm COCOA can perform continual learning without forgetting previous tasks, analyzing its convergence and error performance in sequential task settings.

Contribution

It provides the first analysis of COCOA's ability to handle continual learning in a distributed setting, including closed-form expressions and convergence conditions.

Findings

COCOA can perform continual learning with sequential tasks under certain conditions.

The convergence and error depend on problem dimensions and data assumptions.

Distributed continual learning is feasible with COCOA without access to all past data.

Abstract

We focus on the continual learning problem where the tasks arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the continual learning literature focusing on the centralized setting, we investigate the distributed estimation framework. We consider the well-established distributed learning algorithm COCOA. We derive closed form expressions for the iterations for the overparametrized case. We illustrate the convergence and the error performance of the algorithm based on the over/under-parameterization of the problem. Our results show that depending on the problem dimensions and data generation assumptions, COCOA can perform continual learning over a sequence of tasks, i.e., it can learn a new task without forgetting previously learned tasks, with access only to one task at a time.