Deep Class Incremental Learning from Decentralized Data

TL;DR

This paper introduces a new decentralized class-incremental learning framework (DCID) for handling continuous data inflows across multiple repositories, establishing benchmarks and demonstrating its effectiveness through extensive experiments.

Contribution

It formulates the DCIL problem, creates a benchmark, and proposes the DCID framework with knowledge distillation components for decentralized incremental learning.

Findings

DCID outperforms baseline methods in experiments.

The framework effectively transfers knowledge across local models.

Decentralized learning maintains high accuracy with continuous data inflow.

Abstract

In this paper, we focus on a new and challenging decentralized machine learning paradigm in which there are continuous inflows of data to be addressed and the data are stored in multiple repositories. We initiate the study of data decentralized class-incremental learning (DCIL) by making the following contributions. Firstly, we formulate the DCIL problem and develop the experimental protocol. Secondly, we introduce a paradigm to create a basic decentralized counterpart of typical (centralized) class-incremental learning approaches, and as a result, establish a benchmark for the DCIL study. Thirdly, we further propose a Decentralized Composite knowledge Incremental Distillation framework (DCID) to transfer knowledge from historical models and multiple local sites to the general model continually. DCID consists of three main components namely local class-incremental learning, collaborated knowledge distillation among local models, and aggregated knowledge distillation from local models to the general one. We comprehensively investigate our DCID framework by using different implementations of the three components. Extensive experimental results demonstrate the effectiveness of our DCID framework. The codes of the baseline methods and the proposed DCIL will be released at https://github.com/zxxxxh/DCIL.