Tiny Inference-Time Scaling with Latent Verifiers

TL;DR

This paper introduces VHS, a verifier that operates on hidden states of diffusion models to reduce inference costs and improve efficiency in generative tasks, matching or surpassing large language model verifiers.

Contribution

VHS is a novel verifier that works directly on intermediate hidden states of diffusion models, reducing inference costs while maintaining or improving performance.

Findings

Reduces joint generation-and-verification time by 63.3%.

Cuts compute FLOPs by 51%.

Achieves +2.7% improvement on GenEval at same inference budget.

Abstract

Inference-time scaling has emerged as an effective way to improve generative models at test time by using a verifier to score and select candidate outputs. A common choice is to employ Multimodal Large Language Models (MLLMs) as verifiers, which can improve performance but introduce substantial inference-time cost. Indeed, diffusion pipelines operate in an autoencoder latent space to reduce computation, yet MLLM verifiers still require decoding candidates to pixel space and re-encoding them into the visual embedding space, leading to redundant and costly operations. In this work, we propose Verifier on Hidden States (VHS), a verifier that operates directly on intermediate hidden representations of Diffusion Transformer (DiT) single-step generators. VHS analyzes generator features without decoding to pixel space, thereby reducing the per-candidate verification cost while improving or matching the performance of MLLM-based competitors. We show that, under tiny inference budgets with only a small number of candidates per prompt, VHS enables more efficient inference-time scaling reducing joint generation-and-verification time by 63.3%, compute FLOPs by 51% and VRAM usage by 14.5% with respect to a standard MLLM verifier, achieving a +2.7% improvement on GenEval at the same inference-time budget.