Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs

TL;DR

This paper systematically evaluates prompting strategies for exact computation in LLMs, finding that program generation with external interpreters yields perfect accuracy, unlike standard methods.

Contribution

It introduces a synthetic dataset for controlled evaluation and demonstrates that program-of-Thought prompting with external interpreters outperforms other methods.

Findings

Program-of-Thought achieves perfect accuracy on synthetic tasks.

Standard prompting methods only achieve moderate accuracy.

Training a small domain-specific model yields reliable program generation.

Abstract

Large Language Models (LLMs) have demonstrated strong capabilities in natural language understanding and reasoning. However, their ability to perform exact, deterministic computation remains unclear. In this work, we systematically evaluate multiple prompting strategies, including Chain-of-Thought (CoT), Least-to-Most decomposition, Program-of-Thought (PoT), and Self-Consistency (SC), on tasks requiring precise and error-free outputs, including binary counting, longest substring detection, and arithmetic evaluation. To support this study, we introduce a synthetic dataset with diverse natural language instructions, enabling controlled evaluation of exact computation across multiple task types. Our results show that standard prompting methods achieve only moderate accuracy on sequence-based tasks. CoT provides limited improvement, while Least-to-Most suffers from error accumulation. In contrast, PoT achieves perfect accuracy by generating executable code and delegating computation to an external interpreter. Self-Consistency improves robustness through majority voting, but incurs substantial computational overhead. We further train a small domain-specific model (CodeT5-small) to generate executable programs, which achieves perfect accuracy on held-out synthetic test data across all tasks with minimal training cost. Overall, our findings suggest that LLMs may simulate reasoning patterns rather than reliably perform exact symbolic computation. For deterministic tasks, combining LLMs with external tools or using specialized models provides a more reliable and efficient solution.