Momentum-based Weight Interpolation of Strong Zero-Shot Models for Continual Learning

TL;DR

This paper demonstrates that momentum-based weight interpolation enhances the continual learning capabilities of large zero-shot models, maintaining robustness and improving performance over naive fine-tuning in sequential task adaptation.

Contribution

It introduces a simple momentum-based weight interpolation method that improves continual learning performance of zero-shot models, addressing the limitations of naive fine-tuning.

Findings

Over 4% improvement on standard CL benchmarks

Reduces error close to joint training upper limit

Effective in both memory-free and memory-based settings

Abstract

Large pre-trained, zero-shot capable models have shown considerable success both for standard transfer and adaptation tasks, with particular robustness towards distribution shifts. In addition, subsequent fine-tuning can considerably improve performance on a selected downstream task. However, through naive fine-tuning, these zero-shot models lose their generalizability and robustness towards distribution shifts. This is a particular problem for tasks such as Continual Learning (CL), where continuous adaptation has to be performed as new task distributions are introduced sequentially. In this work, we showcase that where fine-tuning falls short to adapt such zero-shot capable models, simple momentum-based weight interpolation can provide consistent improvements for CL tasks in both memory-free and memory-based settings. In particular, we find improvements of over $+4\%$ on standard CL benchmarks, while reducing the error to the upper limit of jointly training on all tasks at once in parts by more than half, allowing the continual learner to inch closer to the joint training limits.