General and Efficient Steering of Unconditional Diffusion

TL;DR

This paper introduces a fast, gradient-free method for controlling unconditional diffusion models, enabling efficient and accurate semantic steering during image generation without retraining or per-step gradients.

Contribution

It proposes a novel approach using offline guidance signals and transferable concept vectors, identified via Recursive Feature Machine, for efficient and effective diffusion model steering.

Findings

Achieves faster inference compared to gradient-based guidance.

Maintains high-quality, controllable image generation across datasets.

Demonstrates effectiveness on CIFAR-10, ImageNet, and CelebA.

Abstract

Guiding unconditional diffusion models typically requires either retraining with conditional inputs or per-step gradient computations (e.g., classifier-based guidance), both of which incur substantial computational overhead. We present a general recipe for efficiently steering unconditional diffusion {without gradient guidance during inference}, enabling fast controllable generation. Our approach is built on two observations about diffusion model structure: Noise Alignment: even in early, highly corrupted stages, coarse semantic steering is possible using a lightweight, offline-computed guidance signal, avoiding any per-step or per-sample gradients. Transferable concept vectors: a concept direction in activation space once learned transfers across both {timesteps} and {samples}; the same fixed steering vector learned near low noise level remains effective when injected at intermediate noise levels for every generation trajectory, providing refined conditional control with efficiency. Such concept directions can be efficiently and reliably identified via Recursive Feature Machine (RFM), a light-weight backpropagation-free feature learning method. Experiments on CIFAR-10, ImageNet, and CelebA demonstrate improved accuracy/quality over gradient-based guidance, while achieving significant inference speedups.