Steering diffusion models with quadratic rewards: a fine-grained analysis

TL;DR

This paper analyzes the computational complexity of sampling from reward-tilted diffusion models with quadratic rewards, providing efficient algorithms for certain cases and proving intractability for others.

Contribution

It introduces a detailed analysis of quadratic reward tilts in diffusion models, including a new algorithm for low-rank positive-definite cases and complexity results for negative-definite cases.

Findings

Linear quadratic rewards are always efficiently sampleable.

New algorithm based on Hubbard-Stratonovich transform for low-rank positive-definite tilts.

Negative-definite quadratic tilts are computationally intractable even at rank 1.

Abstract

Inference-time algorithms are an emerging paradigm in which pre-trained models are used as subroutines to solve downstream tasks. Such algorithms have been proposed for tasks ranging from inverse problems and guided image generation to reasoning. However, the methods currently deployed in practice are heuristics with a variety of failure modes -- and we have very little understanding of when these heuristics can be efficiently improved. In this paper, we consider the task of sampling from a reward-tilted diffusion model -- that is, sampling from $p^{\star}(x) \propto p(x) \exp(r(x))$ -- given a reward function $r$ and pre-trained diffusion oracle for $p$. We provide a fine-grained analysis of the computational tractability of this task for quadratic rewards $r(x) = x^\top A x + b^\top x$. We show that linear-reward tilts are always efficiently sampleable -- a simple result that seems to have gone unnoticed in the literature. We use this as a building block, along with a conceptually new ingredient -- the Hubbard-Stratonovich transform -- to provide an efficient algorithm for sampling from low-rank positive-definite quadratic tilts, i.e. $r(x) = x^\top A x$ where $A$ is positive-definite and of rank $O(1)$. For negative-definite tilts, i.e. $r(x) = - x^\top A x$ where $A$ is positive-definite, we prove that the problem is intractable even if $A$ is of rank 1 (albeit with exponentially-large entries).