FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model

TL;DR

FreeDoM introduces a training-free, energy-guided conditional diffusion model that leverages pre-trained networks for flexible, low-cost, and effective conditional data generation across multiple domains.

Contribution

It proposes a novel training-free approach using off-the-shelf pre-trained networks to construct energy functions for diverse conditions, broadening application scope.

Findings

Effective across various conditions and data domains

Uses pre-trained networks to guide diffusion without additional training

Demonstrates simplicity, effectiveness, and low cost

Abstract

Recently, conditional diffusion models have gained popularity in numerous applications due to their exceptional generation ability. However, many existing methods are training-required. They need to train a time-dependent classifier or a condition-dependent score estimator, which increases the cost of constructing conditional diffusion models and is inconvenient to transfer across different conditions. Some current works aim to overcome this limitation by proposing training-free solutions, but most can only be applied to a specific category of tasks and not to more general conditions. In this work, we propose a training-Free conditional Diffusion Model (FreeDoM) used for various conditions. Specifically, we leverage off-the-shelf pre-trained networks, such as a face detection model, to construct time-independent energy functions, which guide the generation process without requiring training. Furthermore, because the construction of the energy function is very flexible and adaptable to various conditions, our proposed FreeDoM has a broader range of applications than existing training-free methods. FreeDoM is advantageous in its simplicity, effectiveness, and low cost. Experiments demonstrate that FreeDoM is effective for various conditions and suitable for diffusion models of diverse data domains, including image and latent code domains.