Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment

TL;DR

This paper introduces a self-improving framework for multi-objective alignment of large language models, effectively resolving preference conflicts and achieving Pareto optimality through self-generated responses.

Contribution

It proposes a novel self-improving DPO framework that enables LLMs to generate and select Pareto-optimal responses for better multi-objective alignment.

Findings

Achieves superior Pareto Front compared to baselines

Effectively resolves preference conflicts in data

Demonstrates effectiveness on two datasets

Abstract

Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach. However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses. This results in conflicting optimization directions, hindering the optimization on the Pareto Front. To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts. To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment. Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines. Code is available at https://github.com/zyttt-coder/SIPO.