Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL

TL;DR

This paper introduces a unified framework combining global workspace modeling and multi-level collaborative distillation to improve stability and plasticity in online class-incremental learning under strict memory constraints.

Contribution

It proposes a novel approach that fuses student model parameters into a global workspace and enforces multi-level peer and knowledge distillation, enhancing OCIL performance.

Findings

Significant performance gains on three OCIL benchmarks.

Effective under various memory budgets.

Improved stability and adaptability in models.

Abstract

Online Class-Incremental Learning (OCIL) enables models to learn continuously from non-i.i.d. data streams and samples of the data streams can be seen only once, making it more suitable for real-world scenarios compared to offline learning. However, OCIL faces two key challenges: maintaining model stability under strict memory constraints and ensuring adaptability to new tasks. Under stricter memory constraints, current replay-based methods are less effective. While ensemble methods improve adaptability (plasticity), they often struggle with stability. To overcome these challenges, we propose a novel approach that enhances ensemble learning through a Global Workspace Model (GWM)-a shared, implicit memory that guides the learning of multiple student models. The GWM is formed by fusing the parameters of all students within each training batch, capturing the historical learning trajectory and serving as a dynamic anchor for knowledge consolidation. This fused model is then redistributed periodically to the students to stabilize learning and promote cross-task consistency. In addition, we introduce a multi-level collaborative distillation mechanism. This approach enforces peer-to-peer consistency among students and preserves historical knowledge by aligning each student with the GWM. As a result, student models remain adaptable to new tasks while maintaining previously learned knowledge, striking a better balance between stability and plasticity. Extensive experiments on three standard OCIL benchmarks show that our method delivers significant performance improvement for several OCIL models across various memory budgets.