Learned-Rule-Augmented Large Language Model Evaluators

TL;DR

This paper introduces a rule-augmented evaluation framework for large language models, combining rule distillation and reinforcement learning to improve their generalization and alignment in diverse natural language generation evaluation tasks.

Contribution

It proposes a novel rule-augmented paradigm with rule distillation, Chain-of-Rule guidance, and reinforcement learning to enhance LLM evaluators' effectiveness and generalizability.

Findings

Improved evaluation accuracy across multiple NLG tasks.

Enhanced alignment of LLM evaluators with data and rules.

Demonstrated scalability and versatility of the approach.

Abstract

Large language models (LLMs) are predominantly used as evaluators for natural language generation (NLG) tasks, but their application to broader evaluation scenarios remains limited. In this work, we explore the potential of LLMs as general evaluators across diverse tasks. Although LLM-based evaluators have made progress in different areas, existing methods struggle to generalize due to their reliance on costly, human-designed evaluation principles, which are often misaligned with both annotated data and LLMs' understanding.To address these challenges, we propose a rule-augmented evaluation paradigm. First, we introduce a rule distillation method that automatically extracts scoring rules from data using an LLM-assisted Monte Carlo Tree Search (MCTS), alleviating scalability issues and improving alignment with data. Second, to enable LLMs to effectively apply the learned rules, we propose two strategies: (1) Chain-of-Rule (CoR), which guides LLM to follow distilled rules, and (2) training a rule-augmented LLM evaluator (RuAE) via reinforcement learning, further bridging the gap between rules and LLMs' reasoning. Extensive experiments on diverse tasks demonstrate the effectiveness and generalizability of our approach across various evaluation scenarios.