Sparse Orthogonal Parameters Tuning for Continual Learning

TL;DR

This paper introduces SoTU, a novel method utilizing sparse orthogonal parameters to improve continual learning by effectively merging knowledge from multiple tasks and reducing catastrophic forgetting.

Contribution

The paper proposes SoTU, a new approach that leverages sparse orthogonal parameters for continual learning, demonstrating its effectiveness across various benchmarks.

Findings

SoTU achieves superior performance on multiple CL benchmarks.

It provides optimal feature representations without complex classifiers.

The method effectively merges knowledge from multiple domains.

Abstract

Continual learning methods based on pre-trained models (PTM) have recently gained attention which adapt to successive downstream tasks without catastrophic forgetting. These methods typically refrain from updating the pre-trained parameters and instead employ additional adapters, prompts, and classifiers. In this paper, we from a novel perspective investigate the benefit of sparse orthogonal parameters for continual learning. We found that merging sparse orthogonality of models learned from multiple streaming tasks has great potential in addressing catastrophic forgetting. Leveraging this insight, we propose a novel yet effective method called SoTU (Sparse Orthogonal Parameters TUning). We hypothesize that the effectiveness of SoTU lies in the transformation of knowledge learned from multiple domains into the fusion of orthogonal delta parameters. Experimental evaluations on diverse CL benchmarks demonstrate the effectiveness of the proposed approach. Notably, SoTU achieves optimal feature representation for streaming data without necessitating complex classifier designs, making it a Plug-and-Play solution.