Design of Chain-of-Thought in Math Problem Solving

TL;DR

This paper systematically compares different Chain-of-Thought (CoT) design methods for math problem solving, highlighting the effectiveness of program CoTs and the influence of programming language choices.

Contribution

It introduces a comprehensive comparison of natural language and program CoTs, analyzing programming languages and coding styles, and provides practical guidelines for future CoT design.

Findings

Program CoTs outperform natural language CoTs in math problem solving.

Self-describing program CoTs achieve higher diversity and performance.

Python is more effective than Wolfram Language for program CoTs.

Abstract

Chain-of-Thought (CoT) plays a crucial role in reasoning for math problem solving. We conduct a comprehensive examination of methods for designing CoT, comparing conventional natural language CoT with various program CoTs, including the self-describing program, the comment-describing program, and the non-describing program. Furthermore, we investigate the impact of programming language on program CoTs, comparing Python and Wolfram Language. Through extensive experiments on GSM8K, MATHQA, and SVAMP, we find that program CoTs often have superior effectiveness in math problem solving. Notably, the best performing combination with 30B parameters beats GPT-3.5-turbo by a significant margin. The results show that self-describing program offers greater diversity and thus can generally achieve higher performance. We also find that Python is a better choice of language than Wolfram for program CoTs. The experimental results provide a valuable guideline for future CoT designs that take into account both programming language and coding style for further advancements. Our datasets and code are publicly available.