How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?

TL;DR

This paper introduces CIDM, a continual learning framework for text-to-image diffusion models that effectively learns new concepts without forgetting old ones, enabling flexible and personalized image generation.

Contribution

The paper proposes a novel concept-incremental diffusion model with a consolidation loss and elastic weight aggregation to prevent forgetting and address concept neglect.

Findings

CIDM outperforms existing custom diffusion models in experiments.

The proposed methods effectively mitigate catastrophic forgetting.

The context-controllable synthesis improves personalized image generation.

Abstract

Custom diffusion models (CDMs) have attracted widespread attention due to their astonishing generative ability for personalized concepts. However, most existing CDMs unreasonably assume that personalized concepts are fixed and cannot change over time. Moreover, they heavily suffer from catastrophic forgetting and concept neglect on old personalized concepts when continually learning a series of new concepts. To address these challenges, we propose a novel Concept-Incremental text-to-image Diffusion Model (CIDM), which can resolve catastrophic forgetting and concept neglect to learn new customization tasks in a concept-incremental manner. Specifically, to surmount the catastrophic forgetting of old concepts, we develop a concept consolidation loss and an elastic weight aggregation module. They can explore task-specific and task-shared knowledge during training, and aggregate all low-rank weights of old concepts based on their contributions during inference. Moreover, in order to address concept neglect, we devise a context-controllable synthesis strategy that leverages expressive region features and noise estimation to control the contexts of generated images according to user conditions. Experiments validate that our CIDM surpasses existing custom diffusion models. The source codes are available at https://github.com/JiahuaDong/CIFC.