Reward-free Alignment for Conflicting Objectives

TL;DR

This paper introduces RACO, a reward-free framework for aligning large language models with conflicting objectives, using pairwise preferences and a novel gradient method to improve trade-offs and convergence.

Contribution

It proposes a new reward-free multi-objective alignment method with convergence guarantees and practical heuristics, improving Pareto trade-offs in LLM alignment tasks.

Findings

RACO achieves better Pareto trade-offs than existing methods.

Clipping improves convergence rate in two-objective cases.

Method is effective across multiple LLM families and tasks.

Abstract

Direct alignment methods are increasingly used to align large language models (LLMs) with human preferences. However, many real-world alignment problems involve multiple conflicting objectives, where naive aggregation of preferences can lead to unstable training and poor trade-offs. In particular, weighted loss methods may fail to identify update directions that simultaneously improve all objectives, and existing multi-objective approaches often rely on explicit reward models, introducing additional complexity and distorting user-specified preferences. The contributions of this paper are two-fold. First, we propose a Reward-free Alignment framework for Conflicted Objectives (RACO) that directly leverages pairwise preference data and resolves gradient conflicts via a novel clipped variant of conflict-averse gradient descent. We provide convergence guarantees to Pareto-critical points that respect user-specified objective weights, and further show that clipping can strictly improve convergence rate in the two-objective setting. Second, we improve our method using some heuristics and conduct experiments to demonstrate the compatibility of the proposed framework for LLM alignment. Both qualitative and quantitative evaluations on multi-objective summarization and safety alignment tasks across multiple LLM families (Qwen 3, Llama 3, Gemma 3) show that our method consistently achieves better Pareto trade-offs compared to existing multi-objective alignment baselines.