Create Your World: Lifelong Text-to-Image Diffusion

TL;DR

This paper introduces L2DM, a lifelong text-to-image diffusion model that learns new concepts quickly while retaining previous knowledge, addressing catastrophic forgetting and semantic neglect in image synthesis.

Contribution

The paper proposes a novel lifelong diffusion framework with memory enhancement, concept distillation, and attention modules to improve continual concept learning in text-to-image generation.

Findings

L2DM outperforms state-of-the-art models in qualitative image fidelity.

The model effectively mitigates catastrophic forgetting of previous concepts.

It maintains semantic relevance across diverse continual prompts.

Abstract

Text-to-image generative models can produce diverse high-quality images of concepts with a text prompt, which have demonstrated excellent ability in image generation, image translation, etc. We in this work study the problem of synthesizing instantiations of a use's own concepts in a never-ending manner, i.e., create your world, where the new concepts from user are quickly learned with a few examples. To achieve this goal, we propose a Lifelong text-to-image Diffusion Model (L2DM), which intends to overcome knowledge "catastrophic forgetting" for the past encountered concepts, and semantic "catastrophic neglecting" for one or more concepts in the text prompt. In respect of knowledge "catastrophic forgetting", our L2DM framework devises a task-aware memory enhancement module and a elastic-concept distillation module, which could respectively safeguard the knowledge of both prior concepts and each past personalized concept. When generating images with a user text prompt, the solution to semantic "catastrophic neglecting" is that a concept attention artist module can alleviate the semantic neglecting from concept aspect, and an orthogonal attention module can reduce the semantic binding from attribute aspect. To the end, our model can generate more faithful image across a range of continual text prompts in terms of both qualitative and quantitative metrics, when comparing with the related state-of-the-art models. The code will be released at https://wenqiliang.github.io/.