Reward Incremental Learning in Text-to-Image Generation

TL;DR

This paper introduces Reward Incremental Learning (RIL) for text-to-image diffusion models, addressing the challenge of adapting to multiple objectives over time while mitigating catastrophic forgetting through a novel distillation method.

Contribution

It defines the RIL problem, identifies catastrophic forgetting in diffusion models during incremental learning, and proposes Reward Incremental Distillation (RID) to effectively address these issues.

Findings

RID maintains high-quality image generation across multiple reward tasks

RID significantly reduces catastrophic forgetting in diffusion models

Experimental results validate RID's effectiveness in RIL scenarios

Abstract

The recent success of denoising diffusion models has significantly advanced text-to-image generation. While these large-scale pretrained models show excellent performance in general image synthesis, downstream objectives often require fine-tuning to meet specific criteria such as aesthetics or human preference. Reward gradient-based strategies are promising in this context, yet existing methods are limited to single-reward tasks, restricting their applicability in real-world scenarios that demand adapting to multiple objectives introduced incrementally over time. In this paper, we first define this more realistic and unexplored problem, termed Reward Incremental Learning (RIL), where models are desired to adapt to multiple downstream objectives incrementally. Additionally, while the models adapt to the ever-emerging new objectives, we observe a unique form of catastrophic forgetting in diffusion model fine-tuning, affecting both metric-wise and visual structure-wise image quality. To address this catastrophic forgetting challenge, we propose Reward Incremental Distillation (RID), a method that mitigates forgetting with minimal computational overhead, enabling stable performance across sequential reward tasks. The experimental results demonstrate the efficacy of RID in achieving consistent, high-quality generation in RIL scenarios. The source code of our work will be publicly available upon acceptance.