Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

TL;DR

This paper introduces MOPO, a multi-objective preference optimization algorithm for aligning language models with multiple human objectives, improving upon single-objective methods by directly handling preference data and balancing conflicting goals.

Contribution

The paper presents MOPO, a novel multi-objective preference optimization method that operates on pairwise preference data, enabling better alignment with multiple human objectives without heuristic engineering.

Findings

MOPO approximates the Pareto front on synthetic benchmarks.

MOPO outperforms baselines in real-world human-preference fine-tuning.

Ablation studies show optimization stability and hyperparameter robustness.

Abstract

Post-training of LLMs with RLHF, and subsequently preference optimization algorithms such as DPO, IPO, etc., made a big difference in improving human alignment. However, all such techniques can only work with a single (human) objective. In practice, human users have multiple objectives, such as helpfulness and harmlessness, and there is no natural way to aggregate them into a single objective. In this paper, we address the multi-objective preference-alignment problem, where a policy must optimize several, potentially conflicting, objectives. We introduce the Multi-Objective Preference Optimization (MOPO) algorithm, which frames alignment as a constrained KL-regularized optimization: the primary objective is maximized while secondary objectives are lower-bounded by tunable safety thresholds. Unlike prior work, MOPO operates directly on pairwise preference data, requires no point-wise reward assumption, and avoids heuristic prompt-context engineering. The method recovers policies on the Pareto front whenever the front is attainable; practically, it reduces to simple closed-form iterative updates suitable for large-scale training. On synthetic benchmarks with diverse canonical preference structures, we show that MOPO approximates the Pareto front. When fine-tuning a 1.3B-parameter language model on real-world human-preference datasets, MOPO attains higher rewards and yields policies that Pareto-dominate baselines; ablation studies confirm optimization stability and robustness to hyperparameters.