Hidden Biases in Conditioning Autoregressive Models

TL;DR

This paper investigates the computational complexity of exact inference in autoregressive models, revealing that certain constrained decoding tasks are NP-hard or -hard, highlighting fundamental limitations in neural generation.

Contribution

It formalizes the hardness of exact inference tasks in autoregressive models, providing proofs of NP-hardness and -hardness for constrained decoding and sampling.

Findings

Exact sentence-level MAP decoding is NP-hard.

Exact conditioned normalization is -hard for regular constraints.

Global form constraints make exact decoding computationally intractable.

Abstract

Large language and music models are increasingly used for constrained generation: rhyming lines, fixed meter, inpainting or infilling, positional endings, and other global form requirements. These systems often perform strikingly well, but the induced procedures are usually not exact conditioning of the underlying autoregressive model. This creates a hidden inferential bias, distinct from the better-known notion of bias inherited from the training set: samples are distorted relative to the true constrained distribution, with no generic guarantee of complete coverage of the admissible solution space or of correct conditional probabilities over valid completions. We formalize several exact inference tasks for autoregressive models and prove corresponding hardness results. For succinctly represented autoregressive models whose next-token probabilities are computable in polynomial time, exact sentence-level maximum a posteriori (MAP) decoding is NP-hard. This hardness persists under unary and metrical constraints. On the sampling side, exact conditioned normalization is \#P-hard even for regular constraints such as fixed-length terminal events. Unlike finite-state Markov models, general autoregressive models do not admit a bounded-state dynamic program for these tasks. These results formalize a standard claim in the neural decoding literature: local autoregressive sampling is easy, whereas exact decoding and exact conditioning under global form constraints are computationally intractable in general.