COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Framework

TL;DR

COS-DPO introduces a novel conditioned one-shot fine-tuning framework for multi-objective optimization in large language models, enabling efficient trade-off profiling and post-training control, demonstrated on ranking and alignment tasks.

Contribution

The paper extends Direct Preference Optimization to multi-objective settings with a conditioned approach, enabling efficient one-shot profiling and flexible post-training trade-off adjustments.

Findings

Effective in profiling Pareto frontiers for multi-objective tasks.

Capable of achieving comprehensive trade-offs even after training.

Demonstrated success on ranking and language model alignment tasks.

Abstract

In LLM alignment and many other ML applications, one often faces the Multi-Objective Fine-Tuning (MOFT) problem, i.e., fine-tuning an existing model with datasets labeled w.r.t. different objectives simultaneously. To address the challenge, we propose a Conditioned One-Shot fine-tuning framework (COS-DPO) that extends the Direct Preference Optimization technique, originally developed for efficient LLM alignment with preference data, to accommodate the MOFT settings. By direct conditioning on the weight across auxiliary objectives, our Weight-COS-DPO method enjoys an efficient one-shot training process for profiling the Pareto front and is capable of achieving comprehensive trade-off solutions even in the post-training stage. Based on our theoretical findings on the linear transformation properties of the loss function, we further propose the Temperature-COS-DPO method that augments the temperature parameter to the model input, enhancing the flexibility of post-training control over the trade-offs between the main and auxiliary objectives. We demonstrate the effectiveness and efficiency of the COS-DPO framework through its applications to various tasks, including the Learning-to-Rank (LTR) and LLM alignment tasks, highlighting its viability for large-scale ML deployments.