MarkovScale: Towards Optimal Sequential Scaling at Inference Time

TL;DR

MarkovScale introduces a principled Markov process framework for sequential scaling at inference time, providing theoretical bounds and a system that outperforms existing methods across multiple large language models and benchmarks.

Contribution

It models sequential scaling as a two-state Markov process, deriving optimality conditions and developing a system that balances accuracy and efficiency based on these principles.

Findings

Outperforms state-of-the-art scaling methods across multiple benchmarks

Provides theoretical bounds for accuracy improvements

Demonstrates effectiveness on diverse LLMs and configurations

Abstract

Sequential scaling is a prominent inference-time scaling paradigm, yet its performance improvements are typically modest and not well understood, largely due to the prevalence of heuristic, non-principled approaches that obscure clear optimality bounds. To address this, we propose a principled framework that models sequential scaling as a two-state Markov process. This approach reveals the underlying properties of sequential scaling and yields closed-form solutions for essential aspects, such as the specific conditions under which accuracy is improved and the theoretical upper, neutral, and lower performance bounds. Leveraging this formulation, we develop MarkovScale, a practical system that applies these optimality criteria to achieve a theoretically grounded balance between accuracy and efficiency. Comprehensive experiments across 3 backbone LLMs, 5 benchmarks, and over 20 configurations show that MarkovScale consistently outperforms state-of-the-art parallel and sequential scaling methods, representing a significant step toward optimal and resource-efficient inference in LLMs. The source code will be open upon acceptance at https://open-upon-acceptance.