Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety

TL;DR

Fine-tuning large language models on outlier benign samples can drastically reduce safety, and current defenses are ineffective against this vulnerability, highlighting a critical need for improved safety measures.

Contribution

This study develops a novel attack method by fine-tuning LLMs on outlier benign samples identified through Self-Inf-N, revealing a significant safety risk.

Findings

Fine-tuning on 100 outlier samples severely harms safety.

The attack transfers effectively across different LLM architectures.

Most existing mitigation strategies fail to prevent this safety degradation.

Abstract

Recent studies have uncovered a troubling vulnerability in the fine-tuning stage of large language models (LLMs): even fine-tuning on entirely benign datasets can lead to a significant increase in the harmfulness of LLM outputs. Building on this finding, our red teaming study takes this threat one step further by developing a more effective attack. Specifically, we analyze and identify samples within benign datasets that contribute most to safety degradation, then fine-tune LLMs exclusively on these samples. We approach this problem from an outlier detection perspective and propose Self-Inf-N, to detect and extract outliers for fine-tuning. Our findings reveal that fine-tuning LLMs on 100 outlier samples selected by Self-Inf-N in the benign datasets severely compromises LLM safety alignment. Extensive experiments across seven mainstream LLMs demonstrate that our attack exhibits high transferability across different architectures and remains effective in practical scenarios. Alarmingly, our results indicate that most existing mitigation strategies fail to defend against this attack, underscoring the urgent need for more robust alignment safeguards. Codes are available at https://github.com/GuanZihan/Benign-Samples-Matter.