Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

TL;DR

Pref-CTRL introduces a preference-based training framework for LLM alignment that leverages human preferences and outperforms existing gradient-based editing methods on benchmarks.

Contribution

It proposes a novel multi-objective value function for better alignment with human preferences, improving over prior gradient-based representation editing methods.

Findings

Outperforms RE-Control on two benchmark datasets.

Shows greater generalization on out-of-domain datasets.

Source code available at https://github.com/UTS-nlPUG/pref-ctrl.

Abstract

Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.