LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thought

TL;DR

This paper introduces Pedagogical Chain-of-Thought (PedCoT), a novel prompting strategy inspired by educational theory, to improve LLMs' ability to reliably identify mathematical reasoning mistakes, advancing self-correction methods.

Contribution

The paper presents PedCoT, a new prompting approach based on educational principles, that enhances LLMs' accuracy in detecting mathematical reasoning errors.

Findings

PedCoT significantly outperforms baseline prompts in mistake detection.

The approach enables reliable mathematical mistake identification in zero-shot settings.

Educational theory effectively guides prompt design for complex reasoning tasks.

Abstract

Self-correction is emerging as a promising approach to mitigate the issue of hallucination in Large Language Models (LLMs). To facilitate effective self-correction, recent research has proposed mistake detection as its initial step. However, current literature suggests that LLMs often struggle with reliably identifying reasoning mistakes when using simplistic prompting strategies. To address this challenge, we introduce a unique prompting strategy, termed the Pedagogical Chain-of-Thought (PedCoT), which is specifically designed to guide the identification of reasoning mistakes, particularly mathematical reasoning mistakes. PedCoT consists of pedagogical principles for prompts (PPP) design, two-stage interaction process (TIP) and grounded PedCoT prompts, all inspired by the educational theory of the Bloom Cognitive Model (BCM). We evaluate our approach on two public datasets featuring math problems of varying difficulty levels. The experiments demonstrate that our zero-shot prompting strategy significantly outperforms strong baselines. The proposed method can achieve the goal of reliable mathematical mistake identification and provide a foundation for automatic math answer grading. The results underscore the significance of educational theory, serving as domain knowledge, in guiding prompting strategy design for addressing challenging tasks with LLMs effectively.