Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning

TL;DR

This paper introduces Conditional Language Policy (CLP), a flexible framework for finetuning language models to balance multiple conflicting objectives efficiently without needing multiple models.

Contribution

The paper proposes CLP, a novel method that enables steerable multi-objective finetuning of language models, outperforming existing approaches in Pareto efficiency.

Findings

CLP effectively trades off conflicting objectives at inference.

CLP outperforms existing multi-objective finetuning methods.

CLP does not require multiple models for different trade-offs.

Abstract

Reward-based finetuning is crucial for aligning language policies with intended behaviors (e.g., creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at inference time. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective finetuning.