Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing

TL;DR

This paper identifies vulnerabilities in safety-aligned language models and introduces a novel, learning-free debiasing method called TSDI to improve safety-helpfulness trade-offs during generation.

Contribution

The paper proposes TSDI, a learning-free token-level debiasing technique that enhances safety and helpfulness in language models without retraining.

Findings

TSDI improves safety-helpfulness trade-off in language models.

Safety alignment can cause models to generate negative tokens.

TSDI maintains safety while increasing helpfulness.

Abstract

Safety alignment is an essential research topic for real-world AI applications. Despite the multifaceted nature of safety and trustworthiness in AI, current safety alignment methods often focus on a comprehensive notion of safety. By carefully assessing models from the existing safety-alignment methods, we found that, while they generally improved overall safety performance, they failed to ensure safety in specific categories. Our study first identified the difficulty of eliminating such vulnerabilities without sacrificing the model's helpfulness. We observed that, while smaller KL penalty parameters, increased training iterations, and dataset cleansing can enhance safety, they do not necessarily improve the trade-off between safety and helpfulness. We discovered that safety alignment could even induce undesired effects and result in a model that prefers generating negative tokens leading to rejective responses, regardless of the input context. To address this, we introduced a learning-free method, Token-level Safety-Debiased Inference (TSDI), to estimate and correct this bias during the generation process using randomly constructed prompts. Our experiments demonstrated that our method could enhance the model's helpfulness while maintaining safety, thus improving the trade-off Pareto-front.