LLMs cannot find reasoning errors, but can correct them given the error location

TL;DR

This paper demonstrates that large language models struggle to identify reasoning errors but can effectively correct them when provided with the error location, highlighting a key limitation and potential solution for improving LLM reasoning.

Contribution

The study benchmarks LLMs' mistake-finding abilities, shows correction improves with known error locations, and introduces a classifier for locating mistakes without ground truth labels.

Findings

LLMs have difficulty finding logical mistakes in reasoning tasks.

Providing error location information significantly improves correction performance.

A small classifier trained on out-of-domain data outperforms prompting large models in mistake detection.

Abstract

While self-correction has shown promise in improving LLM outputs in terms of style and quality (e.g. Chen et al., 2023b; Madaan et al., 2023), recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in worse performances overall (Huang et al., 2023). In this paper, we show that poor self-correction performance stems from LLMs' inability to find logical mistakes, rather than their ability to correct a known mistake. Firstly, we benchmark several state-of-the-art LLMs on their mistake-finding ability and demonstrate that they generally struggle with the task, even in highly objective, unambiguous cases. Secondly, we test the correction abilities of LLMs -- separately from mistake finding -- using a backtracking setup that feeds ground truth mistake location information to the model. We show that this boosts downstream task performance across our 5 reasoning tasks, indicating that LLMs' correction abilities are robust. Finally, we show that it is possible to obtain mistake location information without ground truth labels or in-domain training data. We train a small classifier with out-of-domain data, which exhibits stronger mistake-finding performance than prompting a large model. We release our dataset of LLM-generated logical mistakes, BIG-Bench Mistake, to enable further research into locating LLM reasoning mistakes.