Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity

TL;DR

This paper introduces ProFS, a tuning-free, sample-efficient method for reducing toxicity in language models by identifying and removing toxic subspaces, offering a robust alternative to traditional preference-based alignment methods.

Contribution

ProFS provides a novel, theory-grounded, tuning-free approach for model editing that effectively reduces toxicity and is more robust to noisy data compared to DPO.

Findings

ProFS outperforms DPO in sample efficiency.

ProFS demonstrates greater robustness to noisy preference data.

ProFS can be interpreted as a denoised variant of DPO.

Abstract

Recent alignment algorithms such as direct preference optimization (DPO) have been developed to improve the safety of large language models (LLMs) by training these models to match human behaviors exemplified by preference data. However, these methods are both computationally intensive and lacking in controllability and transparency, inhibiting their widespread use. Furthermore, these tuning-based methods require large-scale preference data for training and are susceptible to noisy preference data. In this paper, we introduce a tuning-free alignment alternative, ProFS (Projection Filter for Subspaces), and demonstrate its effectiveness under the use case of toxicity reduction. Grounded on theory from factor analysis, ProFS is a sample-efficient model editing approach that identifies a toxic subspace in the model parameter space and reduces model toxicity by projecting away the detected subspace. The toxic subspace is identified by extracting preference data embeddings from the language model, and removing non-toxic information from these embeddings. We show that ProFS is more sample-efficient than DPO, further showcasing greater robustness to noisy data. Finally, we attempt to connect tuning based alignment with editing, by establishing both theoretical and empirical connections between ProFS and DPO, showing that ProFS can be interpreted as a denoised version of a single DPO step.