Parallel Sampling via Autospeculation

TL;DR

This paper introduces parallel algorithms for sampling in autoregressive and diffusion models, reducing expected sampling time to roughly the square root of sequence length using a novel autospeculation technique.

Contribution

It presents the first parallel algorithms achieving sublinear expected sampling time for these models, with a new autospeculation method inspired by speculative decoding.

Findings

Expected sampling time reduced to n^{1/2}

First parallel speedup for diffusion models in high-accuracy regime

Introduces autospeculation technique for accelerated sampling

Abstract

We present parallel algorithms to accelerate sampling via counting in two settings: any-order autoregressive models and denoising diffusion models. An any-order autoregressive model accesses a target distribution $\mu$ on $[q]^n$ through an oracle that provides conditional marginals, while a denoising diffusion model accesses a target distribution $\mu$ on $\mathbb{R}^n$ through an oracle that provides conditional means under Gaussian noise. Standard sequential sampling algorithms require $\widetilde{O}(n)$ time to produce a sample from $\mu$ in either setting. We show that, by issuing oracle calls in parallel, the expected sampling time can be reduced to $\widetilde{O}(n^{1/2})$. This improves the previous $\widetilde{O}(n^{2/3})$ bound for any-order autoregressive models and yields the first parallel speedup for diffusion models in the high-accuracy regime, under the relatively mild assumption that the support of $\mu$ is bounded. We introduce a novel technique to obtain our results: speculative rejection sampling. This technique leverages an auxiliary ``speculative'' distribution~$\nu$ that approximates~$\mu$ to accelerate sampling. Our technique is inspired by the well-studied ``speculative decoding'' techniques popular in large language models, but differs in key ways. Firstly, we use ``autospeculation,'' namely we build the speculation $\nu$ out of the same oracle that defines~$\mu$. In contrast, speculative decoding typically requires a separate, faster, but potentially less accurate ``draft'' model $\nu$. Secondly, the key differentiating factor in our technique is that we make and accept speculations at a ``sequence'' level rather than at the level of single (or a few) steps. This last fact is key to unlocking our parallel runtime of $\widetilde{O}(n^{1/2})$.