Merge and Bound: Direct Manipulations on Weights for Class Incremental Learning

TL;DR

This paper introduces Merge-and-Bound, a novel weight manipulation method for class incremental learning that reduces forgetting and improves performance by merging model weights across tasks with bounded updates.

Contribution

The paper proposes a new weight merging technique with bounded updates for class incremental learning, enhancing knowledge retention without altering model architecture or objectives.

Findings

Outperforms state-of-the-art CIL methods on standard benchmarks

Effectively reduces catastrophic forgetting

Seamlessly integrates with existing CIL frameworks

Abstract

We present a novel training approach, named Merge-and-Bound (M&B) for Class Incremental Learning (CIL), which directly manipulates model weights in the parameter space for optimization. Our algorithm involves two types of weight merging: inter-task weight merging and intra-task weight merging. Inter-task weight merging unifies previous models by averaging the weights of models from all previous stages. On the other hand, intra-task weight merging facilitates the learning of current task by combining the model parameters within current stage. For reliable weight merging, we also propose a bounded update technique that aims to optimize the target model with minimal cumulative updates and preserve knowledge from previous tasks; this strategy reveals that it is possible to effectively obtain new models near old ones, reducing catastrophic forgetting. M&B is seamlessly integrated into existing CIL methods without modifying architecture components or revising learning objectives. We extensively evaluate our algorithm on standard CIL benchmarks and demonstrate superior performance compared to state-of-the-art methods.