An Empirical Study of Reasoning Steps in Thinking Code LLMs

TL;DR

This empirical study investigates the reasoning processes of large language models in code generation, analyzing their reasoning chains, effectiveness, and failure modes across various tasks to improve understanding and future development.

Contribution

It systematically evaluates reasoning chains in multiple state-of-the-art LLMs, introduces a taxonomy of reasoning issues, and examines how task complexity affects reasoning quality.

Findings

Targeted step increases can improve success rates on some models/tasks.

Modest reductions in reasoning steps often preserve success on standard tasks.

Incompleteness is the primary failure mode, especially on hard problems.

Abstract

Thinking Large Language Models (LLMs) generate explicit intermediate reasoning traces before final answers, potentially improving transparency, interpretability, and solution accuracy for code generation. However, the quality of these reasoning chains remains underexplored. We present a comprehensive empirical study examining the reasoning process and quality of thinking LLMs for code generation. We evaluate six state-of-the-art reasoning LLMs (DeepSeek-R1, OpenAI-o3-mini, Claude-3.7-Sonnet-Thinking, Gemini-2.0-Flash-Thinking, Gemini-2.5-Flash, and Qwen-QwQ) across 100 code generation tasks of varying difficulty from BigCodeBench. We quantify reasoning-chain structure through step counts and verbosity, conduct controlled step-budget adjustments, and perform a 21-participant human evaluation across three dimensions: efficiency, logical correctness, and completeness. Our step-count interventions reveal that targeted step increases can improve resolution rates for certain models/tasks, while modest reductions often preserve success on standard tasks, rarely on hard ones. Through systematic analysis, we develop a reasoning-problematic taxonomy, identifying completeness as the dominant failure mode. Task complexity significantly impacts reasoning quality; hard problems are substantially more prone to incompleteness than standard tasks. Our stability analysis demonstrates that thinking LLMs maintain consistent logical structures across computational effort levels and can self-correct previous errors. This study provides new insights into the strengths and limitations of current thinking LLMs in software engineering.