Self-Error-Instruct: Generalizing from Errors for LLMs Mathematical Reasoning

TL;DR

Self-Error-Instruct (SEI) is a novel framework that enhances large language models' mathematical reasoning by generating and utilizing generalized error-focused training data through iterative self-instruction and error clustering.

Contribution

The paper introduces SEI, a method that synthesizes targeted training data from model errors, improving reasoning performance beyond isolated bad cases.

Findings

Improved accuracy on GSM8K and MATH datasets.

Effective error type clustering enhances data quality.

Iterative fine-tuning boosts reasoning generalization.

Abstract

Although large language models demonstrate strong performance across various domains, they still struggle with numerous bad cases in mathematical reasoning. Previous approaches to learning from errors synthesize training data by solely extrapolating from isolated bad cases, thereby failing to generalize the extensive patterns inherent within these cases. This paper presents Self-Error-Instruct (SEI), a framework that addresses these model weaknesses and synthesizes more generalized targeted training data. Specifically, we explore a target model on two mathematical datasets, GSM8K and MATH, to pinpoint bad cases. Then, we generate error keyphrases for these cases based on the instructor model's (GPT-4o) analysis and identify error types by clustering these keyphrases. Next, we sample a few bad cases during each generation for each identified error type and input them into the instructor model, which synthesizes additional training data using a self-instruct approach. This new data is refined through a one-shot learning process to ensure that only the most effective examples are kept. Finally, we use these curated data to fine-tune the target model, iteratively repeating the process to enhance performance. We apply our framework to various models and observe improvements in their reasoning abilities across both in-domain and out-of-domain mathematics datasets. These results demonstrate the effectiveness of self-error instruction in improving LLMs' mathematical reasoning through error generalization.