Characterizing the adversarial vulnerability of speech self-supervised learning

TL;DR

This paper investigates the adversarial vulnerability of speech self-supervised learning models, revealing significant weaknesses against limited-knowledge attacks and demonstrating attack transferability and imperceptibility.

Contribution

It is the first study to analyze the adversarial robustness of speech SSL models in the context of the SUPERB benchmark, highlighting critical security concerns.

Findings

Speech SSL models are vulnerable to limited-knowledge adversaries.

Adversarial attacks transfer between models, indicating robustness issues.

Crafted attacks are imperceptible to human listeners.

Abstract

A leaderboard named Speech processing Universal PERformance Benchmark (SUPERB), which aims at benchmarking the performance of a shared self-supervised learning (SSL) speech model across various downstream speech tasks with minimal modification of architectures and small amount of data, has fueled the research for speech representation learning. The SUPERB demonstrates speech SSL upstream models improve the performance of various downstream tasks through just minimal adaptation. As the paradigm of the self-supervised learning upstream model followed by downstream tasks arouses more attention in the speech community, characterizing the adversarial robustness of such paradigm is of high priority. In this paper, we make the first attempt to investigate the adversarial vulnerability of such paradigm under the attacks from both zero-knowledge adversaries and limited-knowledge adversaries. The experimental results illustrate that the paradigm proposed by SUPERB is seriously vulnerable to limited-knowledge adversaries, and the attacks generated by zero-knowledge adversaries are with transferability. The XAB test verifies the imperceptibility of crafted adversarial attacks.