MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models

TL;DR

This paper introduces MUSIC, an unsupervised data augmentation method that enhances multi-turn reward models by incorporating multi-turn contrastive signals, leading to improved evaluation of multi-turn conversations in large language models.

Contribution

The paper proposes MUSIC, a novel unsupervised augmentation strategy that synthesizes multi-turn contrastive conversation pairs, significantly improving multi-turn reward model performance.

Findings

MUSIC-augmented RM outperforms baseline methods in multi-turn conversation evaluation.

The approach maintains performance on single-turn benchmarks.

Incorporating multi-turn contrasts is crucial for robust multi-turn reward modeling.

Abstract

Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human evaluation. Multi-turn reward models (RMs) offer a scalable alternative and can provide valuable signals for guiding LLM training. While recent work has advanced multi-turn \textit{training} techniques, effective automated \textit{evaluation} specifically for multi-turn interactions lags behind. We observe that standard preference datasets, typically contrasting responses based only on the final conversational turn, provide insufficient signal to capture the nuances of multi-turn interactions. Instead, we find that incorporating contrasts spanning \textit{multiple} turns is critical for building robust multi-turn RMs. Motivated by this finding, we propose \textbf{MU}lti-\textbf{S}tep \textbf{I}nstruction \textbf{C}ontrast (MUSIC), an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs exhibiting differences across multiple turns. Leveraging MUSIC on the Skywork preference dataset, we train a multi-turn RM based on the Gemma-2-9B-Instruct model. Empirical results demonstrate that our MUSIC-augmented RM outperforms baseline methods, achieving higher alignment with judgments from advanced proprietary LLM judges on multi-turn conversations, crucially, without compromising performance on standard single-turn RM benchmarks.