SPIRIT: Patching Speech Language Models against Jailbreak Attacks

TL;DR

This paper investigates the vulnerability of Speech Language Models to jailbreak attacks and introduces a post-hoc patching method that significantly enhances robustness during inference without retraining.

Contribution

It presents a novel post-hoc defense mechanism for SLMs against adversarial jailbreak attacks, improving security with minimal utility loss and no retraining.

Findings

SLMs are highly vulnerable to jailbreak attacks with up to 100% success rate.

Proposed patching defenses increase robustness to 99%.

Defense effectiveness validated on large-scale SLM benchmarks.

Abstract

Speech Language Models (SLMs) enable natural interactions via spoken instructions, which more effectively capture user intent by detecting nuances in speech. The richer speech signal introduces new security risks compared to text-based models, as adversaries can better bypass safety mechanisms by injecting imperceptible noise to speech. We analyze adversarial attacks and find that SLMs are substantially more vulnerable to jailbreak attacks, which can achieve a perfect 100% attack success rate in some instances. To improve security, we propose post-hoc patching defenses used to intervene during inference by modifying the SLM's activations that improve robustness up to 99% with (i) negligible impact on utility and (ii) without any re-training. We conduct ablation studies to maximize the efficacy of our defenses and improve the utility/security trade-off, validated with large-scale benchmarks unique to SLMs.