Codec-Robust Attacks on Audio LLMs

TL;DR

CodecAttack introduces a novel method for creating robust adversarial perturbations in audio LLMs by manipulating the codec's latent space, bypassing compression defenses and achieving high attack success rates across multiple codecs and models.

Contribution

The paper presents CodecAttack, a new attack optimizing perturbations in the codec's latent space, demonstrating robustness against real-world compression and outperforming waveform-based attacks.

Findings

Achieves 85.5% attack success rate on Opus at moderate bitrates.

Transfers effectively to unseen codecs like MP3 and AAC-LC.

Latent perturbations concentrate below 4kHz, aligning with codec bit allocation.

Abstract

Prior attacks on Audio Large Language Models (Audio LLMs) demonstrated that carefully crafted waveform-domain perturbations can force targeted adversarial outputs. As a defense mechanism against these attacks, real-world codec compression preprocessing has been studied to both detect and remove the perturbations. Yet no existing attack has demonstrated robustness against these compressions. We introduce CodecAttack, which optimizes a perturbation in a neural audio codec's continuous latent space rather than directly perturbing the audio waveform. We show that the codec's compression channel, which discards waveform perturbations, transmits perturbations crafted in its own latent space. To further harden the attack across real-world compression channels, we apply multi-bitrate straight-through Expectation-over-Transformation (EoT), all without modifying the target model. Across three realistic Audio LLM deployment scenarios and three target models, CodecAttack achieves an average 85.5% target-substring attack success rate (ASR) on Opus at moderate bitrates, while the waveform baseline trained with identical EoT hardening does not exceed 26% at any bitrate. The attack transfers to held-out codecs, reaching up to 100% ASR on MP3 and 84% on AAC-LC without retraining. A per-band energy analysis shows that the latent perturbation concentrates below 4kHz, exactly where codecs allocate the most bits, while the waveform baseline spreads into higher frequencies that codecs discard. These results demonstrate that lossy compression is not a reliable defense against adversarial audio and that codec-aware attacks pose a practical threat to deployed Audio LLM systems.