What is in Your Safe Data? Identifying Benign Data that Breaks Safety

TL;DR

This paper investigates how benign data used in fine-tuning Large Language Models can unintentionally compromise safety, identifying specific data patterns and proposing a method to detect and mitigate such risks.

Contribution

It introduces a bi-directional anchoring method to identify benign data likely to degrade safety and analyzes data patterns contributing to jailbreaking in LLMs.

Findings

Fine-tuning on certain benign data increases harmful response likelihood to over 70%

Selected data often appear as lists, bullet points, or math questions

Training on 100 targeted data points significantly impacts safety responses

Abstract

Current Large Language Models (LLMs), even those tuned for safety and alignment, are susceptible to jailbreaking. Some have found that just further fine-tuning an aligned model with benign data (i.e., data without harmful content) surprisingly leads to substantial degradation in safety. We delve into the data-centric aspects of why benign fine-tuning inadvertently contributes to jailbreaking. First, we represent fine-tuning data through two lenses: representation and gradient spaces. Additionally, we propose a bi-directional anchoring method that, during the selection process, prioritizes data points that are close to harmful examples and far from benign ones. Our approach effectively identifies subsets of benign data that are more likely to degrade the model's safety after fine-tuning. Training on just 100 of these seemingly benign datapoints surprisingly leads to the fine-tuned model affirmatively responding to >70% of tested harmful requests, compared to <20% after fine-tuning on randomly selected data. We also observe that the selected data frequently appear as lists, bullet points, or math questions, indicating a systematic pattern in fine-tuning data that contributes to jailbreaking.