LLM-42: Enabling Determinism in LLM Inference with Verified Speculation

TL;DR

LLM-42 introduces a scheduling-based method that achieves deterministic inference in large language models by verifying token outputs with minimal overhead, maintaining high throughput and flexibility.

Contribution

This work presents LLM-42, a novel approach that enforces determinism in LLM inference through verification and rollback, avoiding kernel redesigns and reducing overhead.

Findings

Achieves deterministic outputs with minimal performance impact.

Mostly reuses existing GPU kernels without modification.

Overhead is proportional to the amount of traffic requiring determinism.

Abstract

In LLM inference, the same prompt may yield different outputs across different runs. At the system level, this non-determinism arises from floating-point non-associativity combined with dynamic batching and GPU kernels whose reduction orders vary with batch size. A straightforward way to eliminate non-determinism is to disable dynamic batching during inference, but doing so severely degrades throughput. Another approach is to make kernels batch-invariant; however, this tightly couples determinism to kernel design, requiring new implementations. This coupling also imposes fixed runtime overheads, regardless of how much of the workload actually requires determinism. Inspired by ideas from speculative decoding, we present LLM-42, a scheduling-based approach to enable determinism in LLM inference. Our key observation is that if a sequence is in a consistent state, the next emitted token is likely to be consistent even with dynamic batching. Moreover, most GPU kernels use shape-consistent reductions. Leveraging these insights, LLM-42 decodes tokens using a non-deterministic fast path and enforces determinism via a lightweight verify-rollback loop. The verifier replays candidate tokens under a fixed-shape reduction schedule, commits those that are guaranteed to be consistent across runs, and rolls back those violating determinism. LLM-42 mostly re-uses existing kernels unchanged and incurs overhead only in proportion to the traffic that requires determinism.