Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets

TL;DR

This paper introduces Legend, a cost-effective framework that uses representation engineering to automatically annotate safety margins in preference datasets, enhancing reward modeling and harmless alignment for large language models.

Contribution

Legend is the first framework to leverage a safety direction in embedding space for automatic margin annotation, improving dataset quality without additional training.

Findings

Effective in reward modeling and harmless alignment.

Requires only inference time, no extra training.

Scalable and easy to implement.

Abstract

The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop a dataset involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages representation engineering to annotate preference datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.