Training Data Protection with Compositional Diffusion Models

TL;DR

This paper presents Compartmentalized Diffusion Models (CDM), enabling training on separate data sources, composition at inference, and enhanced data protection, with minimal quality loss and improved alignment in text-to-image tasks.

Contribution

Introduction of CDMs that allow isolated training, flexible composition, and data protection in diffusion models, improving efficiency and privacy without significant performance degradation.

Findings

CDMs achieve within 10% FID of monolithic models on vision datasets.

CDMs enable 8x faster forgetting with minimal FID increase.

CDMs improve alignment (TIFA) by 14.33% in text-to-image generation.

Abstract

We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference time. The individual models can be trained in isolation, at different times, and on different distributions and domains and can be later composed to achieve performance comparable to a paragon model trained on all data simultaneously. Furthermore, each model only contains information about the subset of the data it was exposed to during training, enabling several forms of training data protection. In particular, CDMs enable perfect selective forgetting and continual learning for large-scale diffusion models, allow serving customized models based on the user's access rights. Empirically the quality (FID) of the class-conditional CDMs (8-splits) is within 10% (on fine-grained vision datasets) of a monolithic model (no splits), and allows (8x) faster forgetting compared monolithic model with a maximum FID increase of 1%. When applied to text-to-image generation, CDMs improve alignment (TIFA) by 14.33% over a monolithic model trained on MSCOCO. CDMs also allow determining the importance of a subset of the data (attribution) in generating particular samples, and reduce memorization.