MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

TL;DR

MPO is a post-processing method that efficiently combines existing policies to balance diverse human preferences in language model alignment, reducing costs and improving stability.

Contribution

It introduces a novel framework that aggregates policies through log-linear combination using stochastic mirror descent, avoiding costly retraining.

Findings

MPO achieves balanced performance across diverse preferences.

It outperforms or matches existing models.

It significantly reduces computational costs.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has shown promise in aligning large language models (LLMs). Yet its reliance on a singular reward model often overlooks the diversity of human preferences. Recent approaches address this limitation by leveraging multi-dimensional feedback to fine-tune corresponding reward models and train LLMs using reinforcement learning. However, the process is costly and unstable, especially given the competing and heterogeneous nature of human preferences. In this paper, we propose Mixing Preference Optimization (MPO), a post-processing framework for aggregating single-objective policies as an alternative to both multi-objective RLHF (MORLHF) and MaxMin-RLHF. MPO avoids alignment from scratch. Instead, it log-linearly combines existing policies into a unified one with the weight of each policy computed via a batch stochastic mirror descent. Empirical results demonstrate that MPO achieves balanced performance across diverse preferences, outperforming or matching existing models with significantly reduced computational costs.