SARSteer: Safeguarding Large Audio-Language Models via Safe-Ablated Refusal Steering

TL;DR

SARSteer is a novel inference-time framework that enhances safety in Large Audio-Language Models by improving refusal to harmful queries while maintaining benign responses, addressing key limitations of existing methods.

Contribution

It introduces the first inference-time safety defense for LALMs, combining text-derived refusal steering with safe-space ablation to reduce over-refusal and improve harm mitigation.

Findings

SARSteer significantly increases harmful-query refusal rates.

It maintains high accuracy on benign queries.

The framework is validated through extensive experiments.

Abstract

Large Audio-Language Models (LALMs) are becoming essential as a powerful multimodal backbone for real-world applications. However, recent studies show that audio inputs can more easily elicit harmful responses than text, exposing new risks toward deployment. While safety alignment has made initial advances in LLMs and Large Vision-Language Models (LVLMs), we find that vanilla adaptation of these approaches to LALMs faces two key limitations: 1) LLM-based steering fails under audio input due to the large distributional gap between activations, and 2) prompt-based defenses induce over-refusals on benign-speech queries. To address these challenges, we propose Safe-Ablated Refusal Steering (SARSteer), the first inference-time defense framework for LALMs. Specifically, SARSteer leverages text-derived refusal steering to enforce rejection without manipulating audio inputs and introduces decomposed safe-space ablation to mitigate over-refusal. Extensive experiments demonstrate that SARSteer significantly improves harmful-query refusal while preserving benign responses, establishing a principled step toward safety alignment in LALMs. The codes and constructed datasets are released at https://github.com/linweiii/SARSteer.