Filter-Guided Diffusion for Controllable Image Generation

TL;DR

Filter-Guided Diffusion introduces a fast, flexible method for controllable image generation that improves efficiency and diversity over existing state-of-the-art approaches by leveraging filtering operations during diffusion.

Contribution

It proposes a novel Filter-Guided Diffusion method that enhances control, efficiency, and diversity in diffusion-based image generation using filtering during the process.

Findings

FGD achieves higher structural and semantic quality metrics.

FGD is faster and more memory-efficient than existing methods.

FGD enables localized editing with masks.

Abstract

Recent advances in diffusion-based generative models have shown incredible promise for zero shot image-to-image translation and editing. Most of these approaches work by combining or replacing network-specific features used in the generation of new images with those taken from the inversion of some guide image. Methods of this type are considered the current state-of-the-art in training-free approaches, but have some notable limitations: they tend to be costly in runtime and memory, and often depend on deterministic sampling that limits variation in generated results. We propose Filter-Guided Diffusion (FGD), an alternative approach that leverages fast filtering operations during the diffusion process to support finer control over the strength and frequencies of guidance and can work with non-deterministic samplers to produce greater variety. With its efficiency, FGD can be sampled over multiple seeds and hyperparameters in less time than a single run of other SOTA methods to produce superior results based on structural and semantic metrics. We conduct extensive quantitative and qualitative experiments to evaluate the performance of FGD in translation tasks and also demonstrate its potential in localized editing when used with masks. Project page: https://filterguideddiffusion.github.io/