Code Execution as Grounded Supervision for LLM Reasoning

TL;DR

This paper introduces a scalable approach to generate high-quality reasoning supervision for large language models by leveraging program execution, resulting in improved reasoning abilities and reduced inference token length.

Contribution

The authors propose a novel method that extracts verifiable reasoning traces from code execution to create reliable supervision data for LLMs, bypassing costly annotations.

Findings

Effective transfer of reasoning abilities across tasks

High accuracy in generated reasoning data

Reduced token length during inference

Abstract

Training large language models (LLMs) with chain-of-thought (CoT) supervision has proven effective for enhancing their reasoning abilities. However, obtaining reliable and accurate reasoning supervision remains a significant challenge. We propose a scalable method for generating a high-quality CoT supervision dataset by leveraging the determinism of program execution. Unlike existing reasoning dataset generation methods that rely on costly human annotations or error-prone LLM-generated CoT, our approach extracts verifiable, step-by-step reasoning traces from code execution and transforms them into a natural language CoT reasoning. Experiments on reasoning benchmarks across various domains show that our method effectively equips LLMs with transferable reasoning abilities across diverse tasks. Furthermore, the ablation studies validate that our method produces highly accurate reasoning data and reduces overall token length during inference by reducing meaningless repetition and overthinking.