Forget Less by Learning Together through Concept Consolidation

TL;DR

This paper introduces FL2T, a novel framework for continual learning in Custom Diffusion Models that enables concurrent, order-agnostic concept learning while effectively reducing catastrophic forgetting through inter-concept guidance.

Contribution

The paper proposes a set-invariant inter-concept learning module for improved knowledge retention and transfer in continual learning of diffusion models, addressing limitations of sequential learning approaches.

Findings

Significant improvement in concept retention across three datasets.

At least 2% average gain on CLIP Image Alignment scores.

Effective mitigation of catastrophic forgetting in incremental concept learning.

Abstract

Custom Diffusion Models (CDMs) have gained significant attention due to their remarkable ability to personalize generative processes. However, existing CDMs suffer from catastrophic forgetting when continuously learning new concepts. Most prior works attempt to mitigate this issue under the sequential learning setting with a fixed order of concept inflow and neglect inter-concept interactions. In this paper, we propose a novel framework - Forget Less by Learning Together (FL2T) - that enables concurrent and order-agnostic concept learning while addressing catastrophic forgetting. Specifically, we introduce a set-invariant inter-concept learning module where proxies guide feature selection across concepts, facilitating improved knowledge retention and transfer. By leveraging inter-concept guidance, our approach preserves old concepts while efficiently incorporating new ones. Extensive experiments, across three datasets, demonstrates that our method significantly improves concept retention and mitigates catastrophic forgetting, highlighting the effectiveness of inter-concept catalytic behavior in incremental concept learning of ten tasks with at least 2% gain on average CLIP Image Alignment scores.