MIND: Multi-Task Incremental Network Distillation

TL;DR

MIND is a novel parameter isolation method that significantly improves replay-free class-incremental learning and domain-incremental scenarios, achieving state-of-the-art accuracy on multiple datasets by enhancing knowledge retention and optimizing normalization layers.

Contribution

The paper introduces two distillation procedures and BachNorm layer optimization, advancing replay-free incremental learning with superior performance and ablation analysis.

Findings

+6% accuracy on CIFAR-100/10

+10% accuracy on TinyImageNet/10

up to +40% accuracy in domain-incremental scenarios

Abstract

The recent surge of pervasive devices that generate dynamic data streams has underscored the necessity for learning systems to adapt continually to data distributional shifts. To tackle this challenge, the research community has put forth a spectrum of methodologies, including the demanding pursuit of class-incremental learning without replay data. In this study, we present MIND, a parameter isolation method that aims to significantly enhance the performance of replay-free solutions and achieve state-of-the-art results on several widely studied datasets. Our approach introduces two main contributions: two alternative distillation procedures that significantly improve the efficiency of MIND increasing the accumulated knowledge of each sub-network, and the optimization of the BachNorm layers across tasks inside the sub-networks. Overall, MIND outperforms all the state-of-the-art methods for rehearsal-free Class-Incremental learning (with an increment in classification accuracy of approx. +6% on CIFAR-100/10 and +10% on TinyImageNet/10) reaching up to approx. +40% accuracy in Domain-Incremental scenarios. Moreover, we ablated each contribution to demonstrate its impact on performance improvement. Our results showcase the superior performance of MIND indicating its potential for addressing the challenges posed by Class-incremental and Domain-Incremental learning in resource-constrained environments.