Training Language Model to Critique for Better Refinement

TL;DR

This paper introduces RCO, a new framework for training critique models that focus on generating useful feedback to improve language model responses, demonstrated across multiple tasks.

Contribution

The paper presents RCO, a novel critique training method that optimizes critique utility through refinement signals, improving response quality without direct critique preference assessment.

Findings

RCO outperforms traditional critique methods in multiple tasks.

Critiques generated by RCO lead to more effective response refinements.

The framework enhances LLM evaluation and response improvement processes.

Abstract

Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce \textbf{R}efinement-oriented \textbf{C}ritique \textbf{O}ptimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method's effectiveness in enhancing LLM critique-refinement loops.