Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models

TL;DR

This paper introduces Self-Consistent Internal Rewards (SCIR), a framework that enhances the consistency of internal reward models in language models, leading to improved alignment with human preferences and more reliable self-generated preference data.

Contribution

The paper proposes a novel SCIR framework that enforces consistency among internal reward models during training, significantly improving alignment performance of language models.

Findings

SCIR improves alignment performance over baseline methods.

Enforcing internal reward consistency enhances reward modeling capability.

Selective use of consistent preference data boosts reliability.

Abstract

Aligning Large Language Models (LLMs) with human preferences is crucial for their deployment in real-world applications. Recent advancements in Self-Rewarding Language Models suggest that an LLM can use its internal reward models (such as LLM-as-a-Judge) \cite{yuanself} to generate preference data, improving alignment performance without costly human annotation. However, we find that different internal reward models within the same LLM often generate inconsistent preferences. This inconsistency raises concerns about the reliability of self-generated preference data, hinders overall alignment performance, and highlights the need for further research to ensure reliable and coherent alignment with human preferences. To address this limitation, we propose Self-Consistent Internal Rewards (SCIR), a novel framework designed to enhance consistency among internal reward models during training. In each training step, we collect preference predictions from multiple pre-defined internal reward models and enforce consistency and confidence through an inconsistency penalty mechanism, thereby improving the reliability of these internal reward models. We selectively use data with consistent predictions for preference optimization, ensuring the quality of the preference data. By employing self-consistent internal rewards, our method significantly improves the alignment performance and reward modeling capability of LLMs, outperforming baseline methods by a notable margin.