Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning

TL;DR

This paper introduces a novel data-free continual learning method that employs layer-specific model inversion and feature alignment to synthesize data, enabling effective model updates without access to previous data, especially in large pre-trained models.

Contribution

It proposes Per-layer Model Inversion (PMI) for efficient data synthesis and a feature modeling approach to align synthetic data with real data, improving continual learning performance without data replay.

Findings

Significantly reduces inversion iterations with PMI.

Achieves strong continual learning performance across multiple settings.

Effectively aligns synthetic features with real data using Gaussian modeling.

Abstract

Continual learning (CL) aims to incrementally train a model on a sequence of tasks while retaining performance on prior ones. However, storing and replaying data is often infeasible due to privacy or security constraints and impractical for arbitrary pre-trained models. Data-free CL seeks to update models without access to previous data. Beyond regularization, we employ model inversion to synthesize data from the trained model, enabling replay without storing samples. Yet, model inversion in predictive models faces two challenges: (1) generating inputs solely from compressed output labels causes drift between synthetic and real data, and replaying such data can erode prior knowledge; (2) inversion is computationally expensive since each step backpropagates through the full model. These issues are amplified in large pre-trained models such as CLIP. To improve efficiency, we propose Per-layer Model Inversion (PMI), inspired by faster convergence in single-layer optimization. PMI provides strong initialization for full-model inversion, substantially reducing iterations. To mitigate feature shift, we model class-wise features via Gaussian distributions and contrastive model, ensuring alignment between synthetic and real features. Combining PMI and feature modeling, our approach enables continual learning of new classes by generating pseudo-images from semantic-aware projected features, achieving strong effectiveness and compatibility across multiple CL settings.