Understanding Chain-of-Thought Effectiveness in Code Generation: An Empirical and Information-Theoretic Analysis

TL;DR

This paper systematically investigates how Chain-of-Thought prompting improves code generation in large language models, revealing that structured, high-quality CoT significantly enhances accuracy and efficiency depending on model size and language type.

Contribution

It provides an empirical and information-theoretic analysis of CoT prompting, comparing various paradigms across multiple benchmarks and models, and offers practical guidance for effective CoT use.

Findings

Externally guided CoT outperforms direct generation.

Structured CoT improves Pass@1 by 5-12% with fewer tokens.

High-quality reasoning from strong models yields higher accuracy.

Abstract

Large language models (LLMs) achieve strong performance on code generation, but the mechanisms by which Chain-of-Thought (CoT) prompting helps remain unclear. We present a systematic empirical and information-theoretic study of CoT effectiveness in neural code generation, evaluating five paradigms (Zero-Shot, Zero-Shot CoT, Self-Planning, Structured CoT, Reasoning-CoT) across six Python benchmarks, a multilingual benchmark with 12 programming languages, and six models from 7B to 480B parameters, using conditional mutual information $I(Y;C|X)$ as a conceptual lens. Our results show that externally guided CoT consistently outperforms direct generation, with structured methods improving Pass@1 by 5--12\% on average while using substantially fewer tokens than reflective reasoning, and that CoT benefits depend on language type systems and model capacity. We further find that reasoning \emph{quality} is critical: high-quality structured CoT from strong generators yields significantly higher accuracy than lightweight alternatives with the same template, whereas naive Zero-Shot CoT can even degrade performance. These findings provide practical guidance for choosing CoT strategies based on model capacity, language characteristics, and task complexity.