Enhancing reasoning accuracy in large language models during inference time

TL;DR

This paper investigates inference-time techniques like self-consistency, dual-model agreement, and self-reflection to improve reasoning accuracy of large language models, finding that stochastic decoding significantly enhances performance with minimal overhead.

Contribution

The study provides a systematic comparison of three inference-time strategies for improving LLM reasoning accuracy using Chain-of-Thought prompting.

Findings

Self-consistency with nucleus sampling improves accuracy by 9-15%.

Dual-model agreement enhances reliability in moderate-risk scenarios.

Self-reflection offers limited gains for smaller models.

Abstract

Large Language Models (LLMs) often exhibit strong linguistic abilities while remaining unreliable on multi-step reasoning tasks, particularly when deployed without additional training or fine-tuning. In this work, we study inference-time techniques to improve the reasoning accuracy of LLMs. We systematically evaluate three classes of inference-time strategies: (i) self-consistency via stochastic decoding, where the model is sampled multiple times using controlled temperature and nucleus sampling and the most frequent final answer is selected; (ii) dual-model reasoning agreement, where outputs from two independent models are compared and only consistent reasoning traces are trusted; and (iii) self-reflection, where the model critiques and revises its own reasoning. Across all evaluated methods, we employ Chain-of-Thought (CoT) [1] prompting to elicit explicit intermediate reasoning steps before generating final answers. In this work, we provide a controlled comparative evaluation across three inference-time strategies under identical prompting and verification settings. Our experiments on LLM [2] show that self-consistency with nucleus sampling and controlled temperature value yields the substantial gains, achieving a 9% to 15% absolute improvement in accuracy over greedy single-pass decoding, well-suited for low-risk domains, offering meaningful gains with minimal overhead. The dual-model approach provides additional confirmation for model reasoning steps thus more appropriate for moderate-risk domains, where higher reliability justifies additional compute. Self-reflection offers only marginal improvements, suggesting limited effectiveness for smaller non-reasoning models at inference time.