DiFR: Inference Verification Despite Nondeterminism

TL;DR

This paper introduces DiFR, a method for verifying large language model inference correctness by comparing generated tokens to trusted references, effectively detecting errors, bugs, and quantization effects with minimal overhead.

Contribution

The paper presents Token-DiFR and Activation-DiFR, novel verification schemes that reliably detect inference errors and quantization effects with high accuracy and efficiency.

Findings

Token-DiFR detects quantization with AUC > 0.999 within 300 tokens.

Activation-DiFR detects quantization with AUC > 0.999 using only 2 tokens.

The methods reduce communication overhead by 25-75% compared to existing approaches.

Abstract

As demand for LLM inference grows, it is becoming increasingly important that providers and their customers can verify that inference processes are performed correctly, without errors or tampering. However, re-running the same inference process twice often leads to different results due to benign numerical noise, making it difficult to distinguish legitimate variation from actual problems. To address this problem, we introduce Token-DiFR (Token-Divergence-From-Reference), a method for verifying inference outputs by comparing generated tokens against predictions made by a trusted reference implementation conditioned on the same random seed. Sampling seed synchronization tightly constrains valid outputs, leaving providers minimal room to deviate from correct inference, which allows output tokens themselves to serve as auditable evidence of correctness at zero additional cost to the provider. Token-DiFR reliably identifies sampling errors, simulated bugs, and model quantization, detecting 4-bit quantization with AUC $>$ 0.999 within 300 output tokens. For applications requiring sample-efficient forward-pass verification, we additionally introduce Activation-DiFR, a scheme that uses random orthogonal projections to compress activations into compact fingerprints for subsequent verification. Activation-DiFR detects 4-bit quantization with AUC $>$ 0.999 using just 2 output tokens, while reducing communication overhead by 25-75% relative to existing methods. We release an open-source integration with vLLM to accelerate practical deployment of verifiable inference.