Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

TL;DR

This paper introduces Bi-Factorial Preference Optimization (BFPO), a supervised learning framework that effectively balances safety and helpfulness in language models, outperforming existing methods with less resource use.

Contribution

BFPO re-parameterizes joint safety-helpfulness optimization into a single supervised learning task, reducing resource costs and improving balance in language model fine-tuning.

Findings

BFPO outperforms existing approaches in safety and helpfulness.

BFPO achieves safety levels comparable to human-labor-intensive methods.

BFPO requires less than 10% of the resources used by traditional methods.

Abstract

Fine-tuning large language models (LLMs) on human preferences, typically through reinforcement learning from human feedback (RLHF), has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during fine-tuning remains a critical concern, and mitigating the potential conflicts in safety and helpfulness is costly in RLHF. To address this issue, we propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO), which re-parameterizes a joint RLHF objective of both safety and helpfulness into a single supervised learning objective. In supervised optimization, a labeling function is used to capture the global preferences ranking to balance both safety and helpfulness. To evaluate BFPO, we develop a benchmark that includes comprehensive discriminative and generative tasks for helpfulness and harmlessness. The results indicate that our method significantly outperforms existing approaches in both safety and helpfulness. Moreover, BFPO achieves the same level of safety as methods that heavily rely on human labor with less than 10\% of the computational resources and human prompting and annotation process. The training recipes can be found here: https://github.com/wx-zhang/bfpo.