When Independent Sampling Outperforms Agentic Reasoning

TL;DR

This study compares agent-based reasoning and independent sampling for competitive programming, finding that independent sampling often outperforms agentic reasoning in accuracy-cost tradeoffs under fixed compute budgets.

Contribution

It demonstrates that independent exploration can surpass agentic reasoning in algorithmic tasks, providing empirical results and a budget-allocation analysis.

Findings

k-shot sampling outperforms agent reasoning across models and difficulty levels.

Independent sampling achieves better accuracy-cost tradeoffs.

A theoretical analysis shows optimal budget allocation minimizes log failure likelihood.

Abstract

We study how to allocate inference-time compute for competitive programming under fixed budgets. Evaluating 216 Codeforces problems across Divisions 1-3, we compare agent-based reasoning with repeated independent sampling (k-shot) as a function of both cost and number of model calls. Across models and difficulty levels, k-shot consistently achieves a better accuracy-cost and accuracy-query tradeoff. This gap persists despite prompt caching in agent frameworks, indicating lower per-call effectiveness. Our results show that, for self-contained algorithmic tasks, independent exploration can outperform deeper agentic reasoning under realistic resource constraints. We also provide a budget-allocation analysis when the inference budget is fixed, and prove that a cost-optimal solver minimizes the principled metric log failure likelihood per dollar.