2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision

TL;DR

This paper introduces 2D-DPO, a novel framework that extends preference optimization for LLMs to two dimensions—segments and aspects—using a new dataset, leading to improved alignment with human preferences.

Contribution

The work proposes a 2D preference optimization framework and a new dataset, enabling multi-dimensional feedback to better align LLMs with human preferences.

Findings

2D-DPO outperforms scalar and 1D preference methods on benchmarks.

The dataset HelpSteer-2D captures multi-dimensional human preferences.

Multi-dimensional feedback improves LLM alignment quality.

Abstract

Recent advancements in Direct Preference Optimization (DPO) have significantly enhanced the alignment of Large Language Models (LLMs) with human preferences, owing to its simplicity and effectiveness. However, existing methods typically optimize a scalar score or ranking reward, thereby overlooking the multi-dimensional nature of human preferences. In this work, we propose to extend the preference of DPO to two dimensions: segments and aspects. We first introduce a 2D supervision dataset called HelpSteer-2D. For the segment dimension, we divide the response into sentences and assign scores to each segment. For the aspect dimension, we meticulously design several criteria covering the response quality rubrics. With the 2-dimensional signals as feedback, we develop a 2D-DPO framework, decomposing the overall objective into multi-segment and multi-aspect objectives. Extensive experiments on popular benchmarks demonstrate that 2D-DPO performs better than methods that optimize for scalar or 1-dimensional preferences.