KNN-Diffusion: Image Generation via Large-Scale Retrieval

TL;DR

KNN-Diffusion introduces a retrieval-based approach enabling efficient, text-free image generation and manipulation by leveraging large-scale kNN retrieval with diffusion models, outperforming existing methods on various datasets.

Contribution

The paper presents a novel retrieval-based framework that allows training small diffusion models without text data and enables flexible image generation and editing.

Findings

Achieves state-of-the-art results in text-free image generation.

Enables out-of-distribution image synthesis by database swapping.

Supports local semantic manipulations while maintaining object identity.

Abstract

Recent text-to-image models have achieved impressive results. However, since they require large-scale datasets of text-image pairs, it is impractical to train them on new domains where data is scarce or not labeled. In this work, we propose using large-scale retrieval methods, in particular, efficient k-Nearest-Neighbors (kNN), which offers novel capabilities: (1) training a substantially small and efficient text-to-image diffusion model without any text, (2) generating out-of-distribution images by simply swapping the retrieval database at inference time, and (3) performing text-driven local semantic manipulations while preserving object identity. To demonstrate the robustness of our method, we apply our kNN approach on two state-of-the-art diffusion backbones, and show results on several different datasets. As evaluated by human studies and automatic metrics, our method achieves state-of-the-art results compared to existing approaches that train text-to-image generation models using images only (without paired text data)