Towards stable training of parallel continual learning

TL;DR

This paper proposes Stable Parallel Continual Learning (SPCL), a method to improve training stability in multi-source continual learning by orthogonalizing network parameters and gradients, leading to more reliable learning across tasks.

Contribution

The paper introduces SPCL, a novel approach that enhances stability in parallel continual learning through orthogonality constraints and gradient management techniques.

Findings

SPCL outperforms existing methods in training stability.

Orthogonal constraints improve feature disentanglement.

Gradient orthogonalization reduces conflicts across tasks.

Abstract

Parallel Continual Learning (PCL) tasks investigate the training methods for continual learning with multi-source input, where data from different tasks are learned as they arrive. PCL offers high training efficiency and is well-suited for complex multi-source data systems, such as autonomous vehicles equipped with multiple sensors. However, at any time, multiple tasks need to be trained simultaneously, leading to severe training instability in PCL. This instability manifests during both forward and backward propagation, where features are entangled and gradients are conflict. This paper introduces Stable Parallel Continual Learning (SPCL), a novel approach that enhances the training stability of PCL for both forward and backward propagation. For the forward propagation, we apply Doubly-block Toeplit (DBT) Matrix based orthogonality constraints to network parameters to ensure stable and consistent propagation. For the backward propagation, we employ orthogonal decomposition for gradient management stabilizes backpropagation and mitigates gradient conflicts across tasks. By optimizing gradients by ensuring orthogonality and minimizing the condition number, SPCL effectively stabilizing the gradient descent in complex optimization tasks. Experimental results demonstrate that SPCL outperforms state-of-the-art methjods and achieve better training stability.