EnchTable: Unified Safety Alignment Transfer in Fine-tuned Large Language Models

TL;DR

EnchTable is a framework that transfers safety alignment to fine-tuned large language models, maintaining safety without extensive retraining, and demonstrating robustness against attacks across diverse models and tasks.

Contribution

We introduce EnchTable, a novel NTK-based safety transfer method that preserves safety alignment in various LLMs without retraining, balancing safety and utility effectively.

Findings

EnchTable reduces unsafe outputs across multiple datasets and models.

It outperforms existing methods in safety and utility metrics.

The framework is robust against static and dynamic jailbreaking attacks.

Abstract

Many machine learning models are fine-tuned from large language models (LLMs) to achieve high performance in specialized domains like code generation, biomedical analysis, and mathematical problem solving. However, this fine-tuning process often introduces a critical vulnerability: the systematic degradation of safety alignment, undermining ethical guidelines and increasing the risk of harmful outputs. Addressing this challenge, we introduce EnchTable, a novel framework designed to transfer and maintain safety alignment in downstream LLMs without requiring extensive retraining. EnchTable leverages a Neural Tangent Kernel (NTK)-based safety vector distillation method to decouple safety constraints from task-specific reasoning, ensuring compatibility across diverse model architectures and sizes. Additionally, our interference-aware merging technique effectively balances safety and utility, minimizing performance compromises across various task domains. We implemented a fully functional prototype of EnchTable on three different task domains and three distinct LLM architectures, and evaluated its performance through extensive experiments on eleven diverse datasets, assessing both utility and model safety. Our evaluations include LLMs from different vendors, demonstrating EnchTable's generalization capability. Furthermore, EnchTable exhibits robust resistance to static and dynamic jailbreaking attacks, outperforming vendor-released safety models in mitigating adversarial prompts. Comparative analyses with six parameter modification methods and two inference-time alignment baselines reveal that EnchTable achieves a significantly lower unsafe rate, higher utility score, and universal applicability across different task domains. Additionally, we validate EnchTable can be seamlessly integrated into various deployment pipelines without significant overhead.