Forget Less by Learning Together through Concept Consolidation
Arjun Ramesh Kaushik, Naresh Kumar Devulapally, Vishnu Suresh Lokhande, Nalini Ratha, Venu Govindaraju

TL;DR
This paper introduces FL2T, a novel framework for continual learning in Custom Diffusion Models that enables concurrent, order-agnostic concept learning while effectively reducing catastrophic forgetting through inter-concept guidance.
Contribution
The paper proposes a set-invariant inter-concept learning module for improved knowledge retention and transfer in continual learning of diffusion models, addressing limitations of sequential learning approaches.
Findings
Significant improvement in concept retention across three datasets.
At least 2% average gain on CLIP Image Alignment scores.
Effective mitigation of catastrophic forgetting in incremental concept learning.
Abstract
Custom Diffusion Models (CDMs) have gained significant attention due to their remarkable ability to personalize generative processes. However, existing CDMs suffer from catastrophic forgetting when continuously learning new concepts. Most prior works attempt to mitigate this issue under the sequential learning setting with a fixed order of concept inflow and neglect inter-concept interactions. In this paper, we propose a novel framework - Forget Less by Learning Together (FL2T) - that enables concurrent and order-agnostic concept learning while addressing catastrophic forgetting. Specifically, we introduce a set-invariant inter-concept learning module where proxies guide feature selection across concepts, facilitating improved knowledge retention and transfer. By leveraging inter-concept guidance, our approach preserves old concepts while efficiently incorporating new ones. Extensive…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Generative Adversarial Networks and Image Synthesis · Topic Modeling
