Error Reflection Prompting: Can Large Language Models Successfully Understand Errors?

TL;DR

This paper introduces Error Reflection Prompting (ERP), a novel method that enhances large language models' reasoning by enabling error recognition and correction, thereby improving robustness, interpretability, and reliability in problem-solving.

Contribution

ERP extends Chain-of-thought prompting by incorporating error recognition and correction, allowing models to reflect on and learn from their mistakes, which is a new approach in model reasoning.

Findings

ERP improves reasoning robustness and interpretability.

Automated ERP generation enables scalable error correction.

Models with ERP outperform traditional CoT in error handling.

Abstract

Prompting methods for language models, such as Chain-of-thought (CoT), present intuitive step-by-step processes for problem solving. These methodologies aim to equip models with a better understanding of the correct procedures for addressing a given task. Despite these advancements, CoT lacks the ability of reflection and error correction, potentially causing a model to perpetuate mistakes and errors. Therefore, inspired by the human ability for said tasks, we propose Error Reflection Prompting (ERP) to further enhance reasoning in language models. Building upon CoT, ERP is a method comprised of an incorrect answer, error recognition, and a correct answer. This process enables the model to recognize types of errors and the steps that lead to incorrect answers, allowing the model to better discern which steps to avoid and which to take. The model is able to generate the error outlines itself with automated ERP generation, allowing for error recognition and correction to be integrated into the reasoning chain and produce scalability and reliability in the process. The results demonstrate that ERP serves as a versatile supplement to conventional CoT, ultimately contributing to more robust and capable reasoning abilities along with increased interpretability in how models ultimately reach their errors.