SteeringDiffusion: A Bottlenecked Activation Control Interface for Diffusion Models

TL;DR

SteeringDiffusion introduces a novel, controllable interface for diffusion models that enables smooth, monotonic content-style trade-offs without retraining, outperforming existing methods in controllability and stability.

Contribution

It proposes a bottlenecked activation control interface that maintains the base model's parameters, allowing continuous, runtime adjustment of content-style trade-offs in diffusion models.

Findings

Produces smooth, monotonic content-style trade-offs.

Outperforms LoRA in controllability and stability.

Reveals correlations between intervention magnitude and inversion-stability.

Abstract

We introduce SteeringDiffusion, a bottlenecked activation-level control interface for diffusion models that exposes a smooth, monotonic, and runtime-adjustable control surface over the content--style trade-off. Our method keeps the U-Net backbone frozen and learns a small, prompt-conditioned latent code projected to FiLM/AdaGN-style modulation parameters. A zero-initialized design guarantees exact equivalence to the base model at zero scale, while timestep-aware gating restricts modulation to later denoising stages. A single scalar at inference continuously traverses the control surface without retraining. Across experiments on Stable Diffusion~1.5 and SDXL covering multiple artistic styles, we show that SteeringDiffusion produces smooth and monotonic content--style trade-offs. Under matched parameter budgets, it outperforms LoRA in controllability and stability, while ControlNet and rank-1 adapters do not expose a comparable control surface. We further introduce an inversion-stability diagnostic based on DDIM inversion, used as a post-hoc trajectory probe, which reveals strong correlations with intervention magnitude. These results position \emph{Steering Bottlenecked Explicit Control (S-BEC)} as a practical, general-purpose control interface for frozen diffusion backbones.