Self-Review Framework for Enhancing Instruction Following Capability of LLM

TL;DR

Re5 is a self-evaluation and revision framework that improves large language models' instruction-following abilities efficiently by extracting task components, structural evaluation, and selective revisions, leading to high-quality outputs with minimal supervision.

Contribution

The paper introduces Re5, a novel self-evaluation and revision framework that enhances instruction adherence and output quality in LLMs through structural and constraint-specific evaluations.

Findings

Re5 achieves comparable performance to GPT-4o-mini generated data with less data.

Re5 maintains high response quality with a 64.24% win rate over initial responses.

Re5 enables efficient long-term alignment improvements through iterative refinement.

Abstract

Various techniques have been proposed to improve large language models (LLMs) adherence to formatting and instruction constraints. One of the most effective approaches involves utilizing high-quality data generated by powerful models. However, such models often fail to fully comply with complex instructions in a single generation. To address this limitation, iterative revision methods have been introduced. Nevertheless, as the number of data points and revision iterations increases, the associated monetary costs grow significantly. As a resource-efficient alternative, methods have been proposed that leverage high-performance evaluation tools to compensate for the limited self-evaluation capabilities of open-source LLMs. However, these approaches often lead to a degradation in output quality due to excessive revision. To overcome these challenges, we propose Re5, a self-evaluation and revision framework designed to enhance instruction-following performance while preserving the quality of the generated content. Re5 extracts task and constraint components from user instructions, performs structural evaluations to prevent error accumulation, and applies fine-grained constraint-specific content evaluations followed by selective revisions. This process ensures precise and quality-preserving improvements. The final high-quality outputs are used for alignment tuning, enabling long-term alignment improvements through a data-centric iterative refinement loop. Experimental results demonstrate that Re5 achieves instruction-following performance comparable to models trained on data generated by GPT-4o-mini, a high-performance model, even with a small amount of data while maintaining response quality with a 64.24%-win rate over the non-revised initial responses. These results validate Re5 as an efficient and effective solution for enhancing instruction adherence with minimal external supervision.