Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning

TL;DR

This paper investigates how continual learning techniques can prevent safety degradation in large language models during task adaptation, demonstrating that CL approaches effectively maintain safety across multiple models and tasks.

Contribution

It introduces the application of continual learning methods to preserve safety in LLMs during fine-tuning, showing their effectiveness in mitigating safety risks.

Findings

CL approaches reduce attack success rates compared to standard fine-tuning

DER outperforms other CL methods and baselines in safety preservation

Results generalize across multiple models and downstream tasks

Abstract

The safety alignment of large language models (LLMs) is becoming increasingly important with their democratization. In this paper, we study the safety degradation that comes with adapting LLMs to new tasks. We attribute this safety compromise to catastrophic forgetting and frame the problem of preserving safety when fine-tuning as a continual learning (CL) problem. We consider the fine-tuning-as-a-service setup where the user uploads their data to a service provider to get a customized model that excels on the user's selected task. We adapt several CL approaches from the literature and systematically evaluate their ability to mitigate safety degradation. These include regularization-based, memory-based, and model merging approaches. We consider two scenarios, (1) benign user data and (2) poisoned user data. Our results demonstrate that CL approaches consistently achieve lower attack success rates than standard fine-tuning. Among these, DER outperforms both other CL methods and existing safety-preserving baselines while maintaining task utility. These findings generalize across three downstream tasks (GSM8K, SST2, Code) and three model families (LLaMA2-7B, Mistral-7B, Gemma-2B), establishing CL as a practical solution to preserve safety.