Enhancing AI Safety Through the Fusion of Low Rank Adapters

TL;DR

This paper investigates using Low-Rank Adapter Fusion (LoRA) to reduce harmful responses in instruction fine-tuned large language models, achieving significant safety improvements while maintaining task performance.

Contribution

It introduces LoRA fusion as a novel method to enhance AI safety by combining task and safety adapters, with extensive benchmarking demonstrating its effectiveness.

Findings

42% reduction in harmful responses

Effective safety improvement with minimal performance loss

Observed safety over-correction rejecting safe prompts

Abstract

Instruction fine-tuning of large language models (LLMs) is a powerful method for improving task-specific performance, but it can inadvertently lead to a phenomenon where models generate harmful responses when faced with malicious prompts. In this paper, we explore Low-Rank Adapter Fusion (LoRA) as a means to mitigate these risks while preserving the model's ability to handle diverse instructions effectively. Through an extensive comparative analysis against established baselines using recognized benchmark datasets, we demonstrate a 42\% reduction in the harmfulness rate by leveraging LoRA fusion between a task adapter and a safety adapter, the latter of which is specifically trained on our safety dataset. However, we also observe exaggerated safety behaviour, where the model rejects safe prompts that closely resemble unsafe ones