Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models

TL;DR

This paper introduces retrieval-augmented diffusion models (RDMs) for artistic image synthesis, which improve style control by retrieving style-specific images during training and inference, outperforming traditional prompt-engineering methods.

Contribution

The authors propose a novel retrieval-augmented diffusion approach that allows flexible style specification in image synthesis, surpassing text prompt methods in effectiveness.

Findings

Retrieval-augmented models outperform prompt-engineering in style control

Specialized databases improve style specificity during inference

Open-source code and models are provided for reproducibility

Abstract

Novel architectures have recently improved generative image synthesis leading to excellent visual quality in various tasks. Of particular note is the field of ``AI-Art'', which has seen unprecedented growth with the emergence of powerful multimodal models such as CLIP. By combining speech and image synthesis models, so-called ``prompt-engineering'' has become established, in which carefully selected and composed sentences are used to achieve a certain visual style in the synthesized image. In this note, we present an alternative approach based on retrieval-augmented diffusion models (RDMs). In RDMs, a set of nearest neighbors is retrieved from an external database during training for each training instance, and the diffusion model is conditioned on these informative samples. During inference (sampling), we replace the retrieval database with a more specialized database that contains, for example, only images of a particular visual style. This provides a novel way to prompt a general trained model after training and thereby specify a particular visual style. As shown by our experiments, this approach is superior to specifying the visual style within the text prompt. We open-source code and model weights at https://github.com/CompVis/latent-diffusion .