Imperio: Robust Over-the-Air Adversarial Examples for Automatic Speech Recognition Systems

TL;DR

This paper introduces a novel algorithm for generating robust over-the-air adversarial audio examples that are effective across different room environments without prior room knowledge, leveraging room impulse responses and psychoacoustic hiding techniques.

Contribution

The authors present the first generic over-the-air adversarial example generation method that is robust to room variations and does not require environment-specific adjustments.

Findings

Adversarial examples remain effective across various room configurations.

No line-of-sight between speaker and microphone is necessary for successful attacks.

Psychoacoustic methods can hide adversarial perturbations below human hearing thresholds.

Abstract

Automatic speech recognition (ASR) systems can be fooled via targeted adversarial examples, which induce the ASR to produce arbitrary transcriptions in response to altered audio signals. However, state-of-the-art adversarial examples typically have to be fed into the ASR system directly, and are not successful when played in a room. The few published over-the-air adversarial examples fall into one of three categories: they are either handcrafted examples, they are so conspicuous that human listeners can easily recognize the target transcription once they are alerted to its content, or they require precise information about the room where the attack takes place, and are hence not transferable to other rooms. In this paper, we demonstrate the first algorithm that produces generic adversarial examples, which remain robust in an over-the-air attack that is not adapted to the specific environment. Hence, no prior knowledge of the room characteristics is required. Instead, we use room impulse responses (RIRs) to compute robust adversarial examples for arbitrary room characteristics and employ the ASR system Kaldi to demonstrate the attack. Further, our algorithm can utilize psychoacoustic methods to hide changes of the original audio signal below the human thresholds of hearing. In practical experiments, we show that the adversarial examples work for varying room setups, and that no direct line-of-sight between speaker and microphone is necessary. As a result, an attacker can create inconspicuous adversarial examples for any target transcription and apply these to arbitrary room setups without any prior knowledge.

Tables

Table 1. Table 1. Range of room dimensions for sampling the different distributions. H θ gen subscript 𝐻 subscript 𝜃 gen H_{\theta_{\text{gen}}} describes a generic room, which is used for the generic version of the attack, where we assume the attacker to have no prior knowledge. In case of H θ adp subscript 𝐻 subscript 𝜃 adp H_{\theta_{\text{adp}}} , the distributions are adapted to the lab setup in Figure  7 .

	$b_x$		$b_y$		$b_z$		$T_{60}$
	min	max	min	max	min	max	min	max
$H_{{\theta }_{\text{gen}}}$	2.0 m	15.0 m	2.0 m	15.0 m	2.0 m	5.0 m	0.0 s	1.0 s
$H_{{\theta }_{\text{adp}}}$	6.0 m	10.0 m	5.0 m	9.0 m	3.0 m	5.0 m	0.2 s	0.6 s

Table 2. Table 2. WER, number of successful adversarial examples, and SNRseg for generic over-the-air attacks using H θ gen subscript 𝐻 subscript 𝜃 gen H_{\theta_{\text{gen}}} with speech data for different  M 𝑀 M and varying T 60 subscript 𝑇 60 T_{60} .

	$M=512$		$M=1024$		$M=8192$
	WER	AEs	WER	AEs	WER	AEs
$T_{60}=0.42$ s	42.2 %	5/20	34.9 %	5/20	33.3 %	2/20
$T_{60}=0.51$ s	68.9 %	1/20	56.4 %	2/20	42.0 %	2/20
$T_{60}=0.65$ s	91.6 %	0/20	88.0 %	0/20	68.7 %	2/20
SNRseg	7.6$\pm$6.7 dB		7.7$\pm$6.7 dB		3.2$\pm$6.1 dB

Table 3. Table 3. WER, number of successful adversarial examples, and SNRseg for generic over-the-air attacks using H θ gen subscript 𝐻 subscript 𝜃 gen H_{\theta_{\text{gen}}} and hearing thresholds with speech data for different  M 𝑀 M .

	$M=512$		$M=1024$		$M=8192$
	WER	AEs	WER	AEs	WER	AEs
$T_{60}=0.42$ s	70.0 %	0/20	62.7 %	0/20	69.6 %	2/20
SNRseg	11.5$\pm$5.2 dB		10.4$\pm$6.9 dB		5.5$\pm$4.8 dB

Table 4. Table 4. WER, number of successful adversarial examples, and SNRseg for different audio content for M = 512 𝑀 512 M=512 .

	Music	Speech	Birds
Sucessful AEs	1/20	5/20	0/20
WER	61.1 %	42.2 %	71.7 %
SNRseg	10.7$\pm$2.7 dB	7.6$\pm$6.7 dB	1.2$\pm$3.0 dB

Table 5. Table 5. WER, number of successful adversarial examples, and SNRseg for different audio content for M = 512 𝑀 512 M=512 . Comparing generic over-the-air attacks with adapted over-the-air attacks.

	Music		Speech		Birds
	WER	AEs	WER	AEs	WER	AEs
$H_{{\theta }_{\text{gen}}}$, $h_{\text{gen}}$	61.1 %	1/20	42.2 %	5/20	71.7 %	0/20
$H_{{\theta }_{\text{adp}}}$, $h_{\text{adp}}$	63.2 %	2/20	65.0 %	2/20	84.5 %	2/20
$\Delta$ in WER	+  2.1 %		+ 22.8 %		+ 12.8 %

Table 6. Table 6. WER and number of successful adversarial examples for generic over-the-air attacks with and without direct line-of-sight in varying rooms based on speech data for  M = 8192 𝑀 8192 M=8192 .

	Lecture		Meeting		Office
	Room		Room
	WER	AEs	WER	AEs	WER	AEs
w/   line-of-sight	40.0 %	2/20	55.3 %	1/20	74.0 %	1/20
w/o line-of-sight	71.3 %	0/20	62.0 %	0/20	82.7 %	1/20
$\Delta$ in WER	+  31.3 %		+  6.7 %		+  8,7 %

Table 7. Table 7. Microphone and Speaker positions and the reverberation time for each room in Table  6 .

	$T_{60}$	Microphone	Speaker
			w/ line-of-sight	w/o line-of-sight
Lecture	$0.80$ s	$𝐫=[8.1\text{ m},3.4\text{ m},1.2\text{ m}]$	$𝐬=[11.0\text{ m},3.4\text{ m},1.2\text{ m}]$	$𝐬=[8.9\text{ m},2.2\text{ m},0.0\text{ m}]$
Room
Meeting	$0.74$ s	$𝐫=[3.7\text{ m},5.7\text{ m},1.2\text{ m}]$	$𝐬=[1.8\text{ m},5.7\text{ m},1.2\text{ m}]$	$𝐬=[3.7\text{ m},4.9\text{ m},0.0\text{ m}]$
Room
Office	$0.64$ s	$𝐫=[3.8\text{ m},1.8\text{ m},1.2\text{ m}]$	$𝐬=[1.4\text{ m},4.6\text{ m},1.2\text{ m}]$	$𝐬=[-0.5\text{ m},2.0\text{ m},1.2\text{ m}]$

Equations

$$y=\arg \underset{\tilde{y}}{\max }P(\tilde{y}|x)=f(x),$$

$$x^{\prime }=x+\delta ,\text{such that}f(x)\ne f(x^{\prime }).$$

$$x^{\prime }=\arg \underset{\tilde{x}}{\max }P(y^{\prime }|\tilde{x}).$$

$$x_h=x\ast h.$$

$$\begin{array}{rl}{x}_h(n)=& \sum _{m=n-M+1}^{n}x(m)\cdot h(n-m)\\ & \text{with}n=0,\dots ,N-1,\end{array}$$

$$x^{\prime }=\arg \underset{\tilde{x}}{\max }{\mathbb{E}}_{h\sim H_{\theta }}[P(y^{\prime }|{\tilde{x}}_h)].$$

$$\frac{\partial x_h(n)}{\partial x(m)}=h(n-m)\forall n,m.$$

$$\nabla x=\frac{\partial L(y,y^{\prime })}{\partial f(\chi )}\cdot \frac{\partial f(\chi )}{\partial f_p(x_h)}\cdot \frac{\partial f_p(x_h)}{\partial x_h}\cdot \frac{\partial x_h}{\partial x},$$

$$\nabla x=\frac{\partial L(y,y^{\prime })}{\partial f(\chi )}\cdot \frac{\partial f(\chi )}{\partial f_p(x)}\cdot \frac{\partial f_p(x)}{\partial x}$$

$$WER=100\cdot \frac{\mathcal{L}}{N}=100\cdot \frac{D+I+S}{N}.$$

$$\text{SNRseg(dB)}=\frac{10}{K}\sum _{k=0}^{K-1}{\log }_{10}\frac{{\sum }_{t=Tk}^{Tk+T-1}{x}^2(t)}{{\sum }_{t=Tk}^{Tk+T-1}{\sigma }^2(t)},$$

Full text

In this paper, we demonstrate the first algorithm that produces generic adversarial examples against hybrid ASR systems, which remain robust in an over-the-air attack that is not adapted to the specific environment. Hence, no prior knowledge of the room characteristics is required. Instead, we use room impulse responses (RIRs) to compute robust adversarial examples for arbitrary room characteristics and employ the ASR system Kaldi to demonstrate the attack. Further, our algorithm can utilize psychoacoustic methods to hide changes of the original audio signal below the human thresholds of hearing. In practical experiments, we show that the adversarial examples work for varying room setups, and that no direct line-of-sight between speaker and microphone is necessary. As a result, an attacker can create inconspicuous adversarial examples for any target transcription and apply these to arbitrary room setups without any prior knowledge.

adversarial examples, automatic speech recognition, over-the-air attack

††copyright: acmcopyright††journalyear: 2018††doi: 10.1145/1122445.1122456††conference: ACSAC’20: Annual Computer Security Applications Conference; December 07–11, 2020; Austin, TX††booktitle: ACSAC’20: Annual Computer Security Applications Conference, December 07–11, 2020 Austin, TX††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††journalyear: 2020††copyright: licensedothergov††conference: Annual Computer Security Applications Conference; December 7–11, 2020; Austin, USA††booktitle: Annual Computer Security Applications Conference (ACSAC 2020), December 7–11, 2020, Austin, USA††price: 15.00††doi: 10.1145/3427228.3427276††isbn: 978-1-4503-8858-0/20/12

1. Introduction

In restless dreams I walked alone. Narrow streets of cobblestone. ’Neath the halo of a streetlamp. I turned my collar to the cold and damp. When my eyes were stabbed by the flash of a neon light. That split the night. And touched the sound of silence.

Simon & Garfunkel, The Sound of Silence

Substantial improvements in speech recognition accuracy have been achieved in recent years by using acoustic models based on deep neural networks (DNNs). Nevertheless, current studies suggest that there can be significant differences in the mechanism of artificial neural network algorithms compared to human expectations. This is a very unfortunate situation, as a rogue party can abuse this knowledge to create input data, which leads to inconsistent recognition results, without being noticed (Carlini and Wagner, 2017, 2018). As just one example of such attacks, several recent works have demonstrated that it is possible to fool different kinds of ASR systems into outputting a malicious transcription chosen by the attacker (Carlini and Wagner, 2018; Yuan et al., 2018; Schönherr et al., 2019; Qin et al., 2019; Carlini et al., 2016; Abdullah et al., 2019; Yakura and Sakuma, 2019; Szurley and Kolter, 2019).

The practical implications and real-world impact of the demonstrated attacks are unclear at the moment. On the one hand, earlier work fed the adversarial audio examples directly into the ASR system (Carlini and Wagner, 2018; Schönherr et al., 2019; Yuan et al., 2018), hence ignoring all side effects (e. g., echo or reverberation) of a real-world environment, where the sound is transmitted from a loudspeaker to the input microphone of the recognition engine. On the other hand, some works demonstrated adversarial examples that can be played over-the-air (Carlini et al., 2016; Abdullah et al., 2019; Szurley and Kolter, 2019; Yakura and Sakuma, 2019), but these proof-of-concept attacks are either tailored to a single, static room setup or are hard to reproduce systematically with a proven success rate in a different environment like the attack sketched in CommanderSong (Yuan et al., 2018). Recently and independently, Chen et al. (Chen et al., 2020a) showed a first over-the-air attack. Their attack was evaluated against DeepSpeech (Hannun et al., 2014). In contrast, we are showing an attack against Kaldi, a hybrid ASR system, based on a combination of a DNN-based acoustic model and a subsequent search for the optimal word sequence in a weighted-finite-state-transducer model. This approach is conceptually completely different from the end-to-end approach in DeepSpeech, as used by Chen et al. Hybrid systems such as Kaldi are significant here as they show the best performance on many speech recognition tasks, require comparatively little training material, allow for an easy replacement of task-specific grammars, and are therefore widely adopted in the  industry.

In cases where over-the-air adversarial examples have been used in black-box settings, the target transcription is easily perceived by human listeners, once the intended attack is known (Carlini et al., 2016; Abdullah et al., 2019).

We argue that adversarial examples for ASR systems can only be considered a real threat if the targeted recognition is produced even when the signal is played over the air. Compared to previous attacks, where the manipulated speech signal is fed directly into the ASR system, over-the-air attacks are more challenging, as the transmission over the air significantly alters the signal.

Our key insight that forms the basis of this paper is that this transmission can be modeled as a convolution of the original audio signal with the room impulse response (RIR), which describes the alterations of an acoustic signal by the transmission via loudspeaker to the microphone (see Figure 1 for an illustration), where the RIR depends on various factors (Allen and Berkley, 1979). In practice, it is nearly impossible to estimate an exact RIR without having access to the actual room. Therefore, robust adversarial examples need to take a range of possible RIRs into account to increase the success rate. Nevertheless, we show that for a successful attack, it is not necessary to acquire precise knowledge about the attack setup; instead, a generic adversarial example computed for a large variety of possible rooms is  enough.

Robust Adversarial Examples. The first adversarial audio examples imperceptible to humans, even if they know the target transcription, have been described by Carlini and Wagner (Carlini and Wagner, 2018). Other approaches (Schönherr et al., 2019; Qin et al., 2019) have been successful at embedding most changes below the human threshold of hearing, which makes them much harder to notice. On the downside, none of these attacks were successfully demonstrated when played over the air as the adversarial examples need to be fed directly into the ASR system.

Approaches, which did work over the air, have only been tested in a static setup (i. e., fixed position of speaker and microphone with a fixed distance). Yakura’s and Sakuma’s (Yakura and Sakuma, 2019) approach can hide the target transcription but requires physical access to the room to playback the audio while optimizing the adversarial example, which limits their attack to one very specific room setup and is very time costly. Szurley and Kolter (Szurley and Kolter, 2019) published room-dependent robust adversarial examples, which even worked under constraints given by a psychoacoustic model, describing the human perception of sound. However, their adversarial examples have only been evaluated in an anechoic chamber (i. e., a room specifically designed to absorb reflections). The attack can, therefore, not be used in real-world scenarios, but only in carefully constructed laboratories with properties that are never given in natural environments. In other successful over-the-air attacks, human listeners can easily recognize the target transcription once they are alerted to its content (Carlini et al., 2016; Abdullah et al., 2019). Chen et al. (Chen et al., 2020a) showed a first over-the-air attack against the end-to-end recognition system DeepSpeech (Hannun et al., 2014), relying on a database of measured room transfer functions.

In contrast, our approach is inspired by Athalye et al.’s seminal work: A real-world 3D-printed turtle, which is recognized as a rifle from almost every point of view due to an adversarial perturbation (Athalye et al., 2017). The algorithm for creating this 3D object not only minimizes the distortion for one image, but for all possible projections of a 3D object into a 2D image. We borrow the idea and transfer it to the audio domain, replacing the projections by convolutions with RIRs, thereby hardening the audio adversarial example against the transmission through varying rooms.

Contributions. With Imperio, we introduce the first method to compute generic and robust over-the-air adversarial examples against hybrid ASR systems . We achieve this by utilizing an RIR generator to sample from different room setups. We implement a full, end-to-end attack that works in both cases, with and without psychoacoustic hiding. In either case, we can produce successful robust adversarial examples. With our generic approach, it is possible to induce an arbitrary target transcription in any kind of audio without physical access to the target room.

More specifically, for the simulation, the convolution with the sampled RIR is added as an additional layer to the ASR’s underlying neural network, which enables us to update the original audio signal directly under the constraints given by the simulated RIR. For this purpose, the RIRs are drawn out of a distribution of room setups to simulate the over-the-air attack. Using this approach, adversarial examples are hardened to remain robust in real over-the-air attacks across various room setups. We also show a reduction of the added perturbations based on psychoacoustic hiding (Zwicker and Fastl, 2007), by including hearing thresholds in the backpropagation, as proposed by Schönherr et al. (Schönherr et al., 2019).

We have implemented the proposed algorithm to attack the hybrid DNN-HMM ASR system Kaldi (Povey et al., 2011) under varying room conditions. We demonstrate that generic adversarial examples can be computed that are transferable to different rooms and work without line-of-sight, distances in the range of meters, and even if the microphone records no direct sound but only a reflection. In fact, we even show that our generic approach, using only simulated RIRs, creates more robust adversarial examples compared to real measured examples indicating that no prior knowledge about the attack setup is required for our attack.

In summary, we make the following three key contributions:

• •

Robust Over-The-Air Attack. We propose a generic approach to generate robust over-the-air adversarial examples for DNN-HMM-based ASR systems. The attack uses a DNN convolution layer to simulate the effect of arbitrary RIRs, which allows us to alter the raw audio signal directly.

• •

Psychoacoustics. We show that the attack can be combined with psychoacoustic methods for reducing the perceived distortions.

• •

Performance Analysis. We evaluate the success rate of the adversarial attack and analyze the amount of added perturbation. We investigate the influence of increasing reverberation time, increasing microphone-to-speaker distances, different rooms, and no direct line-of-sight between speaker and microphone.

A demonstration of our Imperio attack is available online at http://imperio.adversarial-attacks.net, where we present several adversarial audio files which have been successfully tested when played over-the-air.

2. Background

In the following, we provide an overview of the ASR system that we used in the attack and describe the general approach to calculate audio adversarial examples. Furthermore, we discuss how room simulations can be performed with the help of RIRs and briefly introduce the necessary background from psychoacoustics as these are used to hide the attack.

2.1. Automatic Speech Recognition

For the demonstration of an end-to-end attack, we chose the open-source speech recognition toolkit Kaldi (Povey et al., 2011), which has been used in previous attacks (Yuan et al., 2018; Schönherr et al., 2019) and is also used in commercial tools like Amazon’s Alexa (Schönherr et al., 2019). In Figure 2, a high-level overview of this system is given. The DNN-HMM-based ASR system can be divided into three parts: the feature extraction, which transforms the raw input data into representative features, the DNN as the acoustic model of the system, and the decoding step, which returns the recognized transcription.

Feature Extraction

For the feature extraction, the raw audio is divided into frames (e. g., $20$ ms long) with a certain overlap (e. g., $10$ ms) between two neighboured frames. For each of these frames, a discrete Fourier transform (DFT) is performed to retrieve a frequency representation of the audio input. Next, the magnitude and the logarithm of the resulting complex signal are calculated. The result is a common representation of audio features in the frequency domain. In Schönherr et al.’s approach, this feature extraction is integrated into the DNN, allowing them to directly modify the raw audio data when computing adversarial examples (see Figure 3 for an illustration).

Acoustic Model DNN

The features described above are used as the input for the acoustic model DNN. Based on these, the DNN calculates a matrix of so-called pseudo-posteriors, which describe the probabilities for each of the phones of the language—English, in this case—being present in each time step $t=1\dots T$.

Decoding

Finally, the pseudo-posteriors are used to calculate the most likely transcription via Viterbi decoding and an HMM-based language model.

This so-called hybrid approach, which realizes speech recognition through a search for the most likely path through a matrix of phone posteriors, is easier to train and still achieves better results in comparison to end-to-end approaches (Lüscher et al., 2019).

2.2. Adversarial Audio Examples

For the calculation of adversarial examples, the ASR system can be described as the function

[TABLE]

mapping an audio signal $x$ to its corresponding, most likely transcription $y$. An adversarial example is generated by modifying the original input

[TABLE]

The added distortions $\delta$ can also be restricted, e. g., via hearing thresholds. In this work, only targeted attacks are considered, where the target transcription $y^{\prime}\stackrel{{\scriptstyle!}}{{=}}f(x^{\prime})$ is defined by the attacker. The optimization can, therefore, be described as

[TABLE]

To calculate robust over-the-air adversarial examples, we base our work on the approach proposed by Schönherr et al. (Schönherr et al., 2019) and similar works. The method can be divided into three steps: forced alignment, gradient descent, and restriction of the perturbations via hearing thresholds.

Forced Alignment

Forced alignment is typically used during the training of the ASR systems when no exact alignments between the audio and the transcription data are available. In our case, we utilize this algorithm to find the best possible alignment of the original audio input and the malicious target transcription.

Gradient Descent

For the attack, the feature extraction is integrated into the DNN, so that the raw audio data can be updated directly via gradient descent. For this purpose, the cross-entropy loss is measured between the target output—the pseudo-posteriors—and the actual DNN output, and is used to compute gradients for the optimization algorithm.

Psychoacoustic Hearing Thresholds

The added noise is restricted to those time-frequency ranges, where noise perceptibility is minimal. For this, we use psychoacoustic hearing thresholds described by Zwicker and Fastl (Zwicker and Fastl, 2007).

2.3. Room Impulse Response

When an audio signal is transmitted through a room, as visualized in Figures 1 and 4, the recorded signal can be approximated by convolving the room’s impulse response $h$ with the original audio signal $x$ as

[TABLE]

Here, the convolution operator $\ast$ is a shorthand notation for the multi-path transmission model

[TABLE]

where $N$ is the length of the audio signal, $M$ the length of the RIR $h$, and all $x(n)$ with $n<0$ are assumed to be zero.

In general, the RIR $h$ depends on the size of the room, the positions of the source and the receiver, and other room characteristics such as the sound reflection properties of the walls, any furniture, people, or other contents of the room. Hence, the audio signal received by the ASR system is never identical to the original audio, and an exact RIR is practically impossible to predict. We describe a possible solution for a sufficient approximation in Section 3.

2.4. Psychoacoustics

Psychoacoustics yields an effective measure of (in-)audibility, which is also helpful for the calculation of inconspicuous audio adversarial examples (Schönherr et al., 2019; Qin et al., 2019). Psychoacoustic hearing thresholds describe how the dependencies between frequencies and across time lead to masking effects in human perception (Zwicker and Fastl, 2007). Probably the best-known example for an application of these effects is found in MP3 compression (ISO, 1993), where the compression algorithm uses empirical hearing thresholds to minimize bandwidth or storage requirements. For this purpose, the original input signal is transformed into a smaller but lossy representation.

For an attack, the psychoacoustic hearing thresholds are used to limit the changes in the audio signal to time-frequency-ranges, where the added perturbations are not, or barely, perceptible by humans. To calculate the hearing thresholds, we use the approach described by Schönherr et al. (Schönherr et al., 2019).

3. Over-the-Air Adversarial Examples

Our goal is to compute robust audio adversarial examples, which still work after transmission from a loudspeaker. For this, we simulate different RIRs and employ an iterative algorithm to compute adversarial examples robust against signal modifications that are incurred during playback in a room.

3.1. Threat Model

Throughout the rest of this paper, we consider the following threat model similar to prior work in this area. We assume a white-box attack, where the adversary knows the internals of the ASR system, including all its model parameters. This requirement is in line with prior work on this topic (Schönherr et al., 2019; Carlini and Wagner, 2018; Yuan et al., 2018). Using this knowledge, the attacker generates malicious audio samples offline before the actual attack takes place, i. e., the attacker exploits the ASR system to create an audio file that produces the desired recognition result, which is then played via a loudspeaker. Additionally, we assume that the trained ASR system, including the DNN, remains unchanged over time. Finally, we assume that the adversarial examples are played over the air. Note that we only consider targeted attacks, where the target transcription is predefined (i.e., the adversary chooses the target transcription). Finally, we assume a threat model where a potential attacker can run an extensive search. Specifically, the attacker is able to calculate a batch of potential adversarial examples and select those examples that are especially robust.

3.2. Room Impulse Response Simulator

To simulate RIRs, we use the AudioLabs implementation based on the image method from Allen and Berkley (Allen and Berkley, 1979). The simulator takes as input the room dimensions, the reverberation time $T_{60}$, and the position of source and receiver and approximates the corresponding RIR for the given parameters.

For our attack, we model cuboid-shaped rooms, which can be described by their length $b_{x}$, width $b_{y}$, height $b_{z}$ defined as $\mathbf{b}=[b_{x},b_{y},b_{z}]$. In addition to this, we model the three-dimensional source position $\mathbf{s}=[s_{x},s_{y},s_{z}]$, receiver position $\mathbf{r}=[r_{x},r_{y},r_{z}]$, and the reverberation time $T_{60}$, which is a standard measure for the audio decay time, defined as the time it takes for the sound pressure level to reduce by $60$ dB. This results in ten freely selectable parameters. All parameters are also sketched in Figure 4. Even though this might seem like an overly simple model, we show that the computed adversarial examples are indeed robust for real rooms that are more  complex.

In order to sample random RIRs, we interpret these ten parameters to be random variables. We draw each value from a uniform distribution between a minimum and a maximum allowed value. For the room size and for $T_{60}$, the minimum and the maximum values can be chosen arbitrarily and are thus selected first. After those parameters are drawn, the ranges for source and receiver positions are drawn to guarantee that the source and the receiver are located inside the room.

To simplify the notation, we use the 10-dimensional parameter vector $\theta$ in the following to describe all of these parameters. The RIR $h$ can be considered as a sample of the distribution $H_{\theta}$. An example of a simulated RIR in the time and the frequency domain is shown in Figure 5.

3.3. Robust Audio Adversarial Examples

Unlike earlier approaches that feed adversarial examples directly into the ASR system (Schönherr et al., 2019; Carlini and Wagner, 2018), we explicitly include characteristics of the room, in the form of RIRs, in the optimization problem. This hardens the adversarial examples to remain functional in an over-the-air attack.

For the attack, we therefore extend the optimization criterion given in (3) by

[TABLE]

This approach is derived from the Expectation Over Transformation (EOT) approach in the visual domain, where it is used to consider different two- and three-dimensional transformations, which leads to robust real-world adversarial examples (Athalye et al., 2017). In our case, instead of visual transformations, we use the convolution with RIRs, drawn from $H_{\theta}$, to maximize the expectation over varying RIRs, as shown in Equation (6).

For the implementation, we use a DNN that already has been augmented to include the feature extraction and prepend an additional layer to the DNN. This layer simulates the convolution from the input audio file with the RIR $h$ to model the transmission through the room. Integrating this convolution as an additional layer allows us to apply gradient descent directly to the raw audio signal. A schematic overview of the proposed DNN is given in Figure 6. The first part (“Convolution”) describes the convolution with the RIR $h$. Note that the RIR simulation layer is only used for the calculation of adversarial examples and removed during testing, as the actual RIR will then act during the transmission over the air. The center and right part (“Feature extraction” and “DNN”) show the feature extraction and the acoustic model DNN, which is used to obtain the pseudo-posteriors for the decoding stage.

The inclusion of the convolution as a layer in the DNN requires it to be differentiable. Using (5), the derivative can be written as

[TABLE]

This can be integrated for the calculation of the gradient $\nabla x$ as

[TABLE]

where the function $f_{p}(\cdot)$ describes the feature extraction. This is an extension of the approach in (Schönherr et al., 2019), where

[TABLE]

is defined for the calculation of adversarial examples via gradient descent using the objective function $L(\cdot)$.

3.4. Over-the-Air Adversarial Examples

To assess the robustness of the hardened over-the-air adversarial attack, the adversarial examples $x^{\prime}$ have to be played back via a loudspeaker, and the recorded audio signals are used to determine the accuracy. For the calculation, we implemented the optimization as defined in (6), by sampling a new RIR $h$ after every set of $Q$ gradient descent iterations. This simulates different rooms and recording conditions. Therefore, the generated adversarial example depends on the distribution $H_{\theta}$, from which the RIR $h$ is drawn. After each gradient descent step, the audio signal $x^{\prime}$ is updated via the calculated gradient $\nabla x$ at the learning rate $\alpha$.

The total maximum number of iterations is limited to at most $G$ iterations. However, if a successful robust adversarial example is created before the maximum number of iterations is reached, the algorithm does not need to continue. To efficiently calculate adversarial examples, we use an RIR $h_{\text{test}}$ to simulate the over-the-air scenario during the calculation to verify whether the example has already achieved over-the-air robustness. This RIR is only used for verification and can, for example, be drawn out of $H_{\theta}$ once at the beginning of the algorithm.

The entire approach is summarized with Algorithm 1. As can be seen, the psychoacoustic hearing thresholds $\Phi$ are optionally used during the gradient descent to limit modifications of the signal to those time-frequency ranges, where they are (mostly) imperceptible. Here, $\text{DNN}_{0}$ describes the augmented DNN (“Feature extraction” and “DNN”) in Figure 3 without the RIR simulation since, for the algorithm, this is replaced by the simulated RIR $h_{\text{test}}$.

4. Experimental Evaluation

In the following, we evaluate the performance of the proposed algorithm for adversarial examples played over-the-air and compare the performance for varying reverberation times, distances, and adversarial examples restricted by psychoacoustic hearing thresholds. Additionally, we compare the generic approach with an adapted version of the attack where an attacker has prior knowledge of the target room. Finally, we measure the changes of generic adversarial examples replayed in different rooms and, even if there is no direct line-of-sight between the microphone and the speaker.

For a practical demonstration of the attack, exemplary adversarial examples are available online at http://imperio.adversarial-attacks.net.

4.1. Metrics

We use the following standard measures to assess the quality of the computed adversarial examples.

4.1.1. Word Error Rate

To measure performance, we use the word error rate (WER) with respect to the target transcription. For its computation, the standard metric for this purpose, the Levenshtein distance (Navarro, 2001) $\mathcal{L}$, is used, summing up the number of deleted $D$, inserted $I$, and substituted $S$ words for the best possible alignment between target text and recognition output. The Levenshtein distance is finally divided by the total number of words $N$ to obtain

[TABLE]

For a real attack, an adversarial example can only be considered successful if a WER of [math] % is achieved (i.e., the hypothesis of the system matches with the attacker chosen target transcription).

4.1.2. Segmental Signal-to-Noise Ratio

The segmental signal-to-noise ratio (SNRseg) measures the amount of noise $\sigma$ added from an attacker to the original signal $x$ and is computed as

[TABLE]

where $T$ is the segment length and $K$ the number of segments. Thus, the higher the SNRseg, the less noise was added.

In contrast to the signal-to-noise ratio (SNR), the SNRseg (Voran and Sholl, 1995) is computed frame-wise and gives a better assessment of an audio signal if the original and the added noise are aligned (Yang, 1999) as it is the case in our experiments.

4.2. Calculation Time

All experiments were performed on a machine with an Intel Xeon Gold 6130 CPU and 128 GB of DDR4 memory.

For our experiments, we limit the maximum number of iterations to $2000$ since in every iteration more distortions are added to the audio file, which decreases the audio quality of the adversarial examples. Also, this number is sufficient for the attack to converge, as can be seen in Figure 8, where the WER is plotted as a function of the maximum numbers of iterations $G$.

Computing an adversarial example for $10$ seconds of audio with the maximum number of $G=2000$ iterations and $M=512$ takes about $80$ minutes. Note that the computation for a single audio file is limited by the single-core performance of the machine, and the attack is fully parallelizable for multiple audio files.

4.3. Over-the-Air Attacks

We evaluate the attack for the lab setup as shown in Figure 7. The approximate dimensions of this room are $\mathbf{b}_{\text{real}}\approx[8\text{\,m},7\text{\,m},2.8\text{\,m}]$ and the positions of the loudspeaker and the microphone are $\mathbf{s}_{\text{real}}=[3.9\text{\,m},3.4\text{\,m},1.2\text{\,m}]$ and $\mathbf{r}_{\text{real}}=[1.4\text{\,m},1.8\text{\,m},1.2\text{\,m}]$, respectively.

We compute all adversarial examples with Algorithm 1. Based on preliminary experiments, we set $G=2000$ and $Q=10$. For the distributions $H_{\theta}$, we used two different versions, shown in Table 1. $H_{\theta_{\text{gen}}}$ describes a generic room, while $H_{\theta_{\text{adp}}}$ is used as an approximation to reassemble the real room from Figure 7. If not specified otherwise, $h_{\text{test}}$, which is used for testing during the attack, is drawn once at the beginning of the algorithm from the same distribution $H_{\theta}$.

The WER is measured for the recorded adversarial examples after playing them via loudspeaker. The SNRseg is calculated after applying a measured RIR $h_{\text{real}}$ to both the original signal and the adversarial example. We chose this approach since it corresponds to the actual signal perceived by human listeners if the adversarial examples are played over the air.

For all cases, we calculated $20$ adversarial examples. In some cases, the audio samples clipped too much (exceeded the maximum defined value of the audio, after the addition of the adversarial distortion). As it would not be possible to replay those examples, we removed them from the evaluation of the real over-the-air attack. Each of the remaining adversarial examples were played five times, and we reported the number of adversarial examples that could be transcribed with 0 % WER.

4.3.1. Generic Over-the-Air Attack

First, we evaluate the attack under the assumption that the attacker has no prior knowledge about the attack setup. Specifically, we use $H_{\theta_{\text{gen}}}$ and calculate adversarial examples for different reverberation times $T_{60}$ and varying length $M$ of RIRs. $M$ describes how many past sampling values are considered, and the larger the reverberation time, the more important are the past sampling values. We assume that, especially in setups with high reverberation times $T_{60}$, a larger $M$ will result in more robust adversarial examples, as it is a better match to the real-world conditions.

For the experiments, we used the variable-acoustics lab in Figure 7 to adjust the reverberation time and tested three versions of the RIR length $M=512$, $M=1024$, and $M=8192$ for speech data. The results in Table 2 confirm the above assumption: for $M=8192$, we can obtain the best WERs, especially for the longer reverberation times. Note that even if the WER seems to be high, for an attacker, it is sufficient to play one successful adversarial example with [math] % WER, which is also in line with the definition in Section 3.1 and, in fact possible. The SNRseg decreases with increasing values for $M$, which indicates that more noise needs to be added to these adversarial examples. However, the calculation time of the adversarial examples increases by the factor of four from $M=512$ to $M=8192$.

4.3.2. Hearing Thresholds

To measure the impact of hearing thresholds, we conducted the same experiments as for Table 2 with $T_{60}=0.42$ and hearing thresholds. The results are shown in Table 3. Compared to the version without hearing thresholds, the WER and the total number of successful adversarial examples decrease. Nevertheless, it was possible to find successful adversarial examples for $M=8192$. At the same time, the SNRseg has improved values. Additionally, the SNRseg measures any added noise, not only the perceptible noise components. Therefore, the perceptible noise is even lower than the SNRseg would suggest for the versions where hearing thresholds are used.

4.3.3. Distance between Speaker and Microphone

In Figure 9, we measured the effect of an increasing distance between the microphone and loudspeaker. We used the shortest reverberation time $T_{60}=0.42$ s and varied the distance from $1$ m to $6$ m for $M=8192$ with and without hearing thresholds.

In general, we find that the WER increases with increasing distance. Nevertheless, starting from a distance of approximately $2$ m, the WER does not increase as rapidly as for smaller distances if we use hearing thresholds. In cases where no hearing thresholds are used, the WER even decreases for larger distances.

4.3.4. Varying Audio Content

In Table 4, we evaluated the effect of varying audio contents of the original audio samples. For this, we used speech, music, and bird chirping data. Using speech audio samples for the attack results in the best WERs.

The average SNRseg indicates that most distortions have to be added to the original audio samples for bird chirping while we achieve better results for music and speech data.

4.3.5. Adaptive Attack

In the following, we compare the generic attack, where the attacker has no prior knowledge about the attack setup, with an adapted version of the attack. Note that the generic attack is the more powerful attack compared to the adapted version, as it requires no access nor any information about the room where the attack is conducted.

For the evaluation, we used $H_{\theta_{\text{adp}}}$ and $H_{\theta_{\text{gen}}}$ in Table 1, combined with a measured RIR $h_{\text{real}}$ and a simulated RIR $h_{\text{gen}}$ for $h_{\text{test}}$ in Algorithm 1. $h_{\text{gen}}$ was drawn once at the beginning of the algorithm from the same distribution, described via $H_{\theta_{\text{gen}}}$. $h_{\text{real}}$ is a measured RIR, obtained from the recording setup that is actually used during the attack. Consequently, the version with $H_{\theta_{\text{gen}}}$ and $h_{\text{gen}}$ does not use any prior knowledge of the room or the recording setup, while the version with $H_{\theta_{\text{adp}}}$ and $h_{\text{real}}$ is tailored to the room. Surprisingly, the generic version clearly outperforms the adapted versions ($H_{\theta_{\text{adp}}}$, $h_{\text{real}}$) in Table 5, and we were able to find fully successful adversarial examples for those cases, i. e., adversarial examples with a WER of [math] %.

As a consequence, for an attacker, it is not only unnecessary to acquire prior knowledge about the room characteristics, but the likelihood of success is even higher if a generic attack is chosen.

4.3.6. Varying Room Conditions

To evaluate the adversarial examples in varying rooms, we chose three rooms of differing sizes: a lecture room with approximately $77$ m2, a meeting room with approximately $38$ m2, and an office with approximately $31$ m2. Layout plans of the rooms are shown in Appendix A, including positions of the speaker and microphone for all recording setups and the measured reverberation time.

Direct Line-of-Sight Attack

The first attacks were conducted with a direct line-of-sight between the microphone and the speaker. The results are shown in Table 6. Even though the results vary depending on the room, the WERs remain approximately in the same range as the experiments with varying reverberation times in Table 2 would indicate. Surprisingly, the room with the highest reverberation time, the lecture room, actually gave the best results.

Overall, the results show that our generic adversarial examples remain robust for different kinds of rooms and setups and that it is sufficient to calculate one version of an adversarial example to cover a wide range of rooms (i.e., the attack is transferable).

No Line-of-Sight Attack

For the rooms in Table 6, we also performed experiments where no line-of-sight between the microphone and the speaker exists by blocking the direct over-the-air connection with different kinds of furniture. As a consequence, not the direct sound, but a reflected version of the audio is recorded. An implication is that these attacks could be carried out without being visible to people in the vicinity of the ASR input microphone. We tested different scenarios for our setup: In the lecture and the meeting room, the source and the receiver were separated by a table by simply placing the speaker under the table. In the office, the speaker was even placed outside the room. For this recording setup, the door between the rooms was left open. For all other setups, the doors of the respective rooms were closed. A detailed description of the no line-of-sight setups is given in Appendix A.

In cases where no line-of-sight exists, the distortions that occur through the transmission can be considered more complex, and consequentially, a prediction of the recorded audio signal is hard. A blocked line-of-sight will most likely decrease the WER, but it is in general possible to find adversarial examples with [math] % WER, even where the source was placed outside the room. This again shows that our generic version of the attack can successfully model a wide range of acoustic environments simultaneously, without any prior knowledge about the room setup.

5. Related Work

In addition to the prior work that we have already discussed, we want to provide a broader and more detailed overview of related work in the following.

Generally speaking, adversarial attacks on ASR systems focus either on hiding a target transcription (Carlini et al., 2016; Abdullah et al., 2019) or on obfuscating the original transcription (Cisse et al., 2017). Almost all previous works on attacks against ASR systems did not focus on real-world attacks (Carlini et al., 2016; Zhang et al., 2017) or were only successful for simulated over-the-air attacks (Qin et al., 2019).

Carlini et al. (Carlini et al., 2016) have shown that targeted attacks against HMM-only ASR systems are possible. They use an inverse feature extraction to create adversarial audio samples. However, the resulting audio samples are not intelligible by humans in most cases and may be considered as noise, but may make thoughtful listeners suspicious once they are alerted to its hidden voice command. An approach to overcome this limitation was proposed by Zhang et al. (Zhang et al., 2017). They have shown that an adversary can hide a transcription by utilizing non-linearities of microphones to modulate the baseband audio signals with ultrasound above 20 kHz, which they inject into the environment. The main downside of this attack, hence, is that the attacker needs to place an ultrasound transmitter in the vicinity of the voice-controlled system under attack and that the attacker needs to retrieve information from the audio signal, recorded with the specific microphone, which is costly in practice and tailors the attack to one specific setup. Song and Mittael (Song and Mittal, 2017) and Roy et al. (Roy et al., 2017) introduced similar ultrasound-based attacks that are not adversarial examples, but rather interact with the ASR system in a frequency range inaudible to humans. Nevertheless, for the attack hours of audio recordings are required to adjust the attack to the setup (Song and Mittal, 2017) or specially designed speakers are necessary (Roy et al., 2017).

Carlini and Wagner (Carlini and Wagner, 2018) published a general targeted attack on ASR systems using CTC-loss. The attacker creates the optimal attack via gradient-descent-based minimization (Carlini and Wagner, 2017) (similar to previous adversarial attacks on image classification), but the adversarial examples are fed directly into the recognizer. CommanderSong (Yuan et al., 2018) is evaluated against Kaldi and uses backpropagation to find an adversarial example. However, the very limited and non-systematic over-the-air attack highly depends on the speakers and recording devices, as the attack parameters have to be adjusted, especially for these components. Yakura and Sakuma (Yakura and Sakuma, 2019) published a technical report, which describes an algorithm to create over-the-air robust adversarial examples, but with the limitation that it is necessary to have physical access to the room where the attack takes place. Also, they did not evaluate their room-dependent results for varying room conditions and were unable to create generic adversarial examples systematically. Concurrently, Szuley and Kolter (Szurley and Kolter, 2019) also published a work on room-dependent robust adversarial examples, which worked under constraints given by a psychoacoustic model. However, their adversarial examples only work in an anechoic chamber, a room specifically designed to eliminate the effect of an RIR. The attack can, therefore, not be compared with a real-world scenario, as the anechoic chamber effectively reproduces the effect of directly feeding the attack into the ASR system, which is never given in real room environments. Li et al. (Li et al., 2019) published a work to obfuscate Amazon’s Alexa wake word via specifically crafted music. However, their approach was not successful at creating targeted adversarial examples that work over the air.

Alzantot et al. (Alzantot et al., 2018) proposed a black-box attack, which does not require knowledge about the model. For this, the authors use a genetic algorithm to create their adversarial examples for a keyword spotting system, which differs from general speech recognition due to a much simpler architecture and far fewer possible recognition outcomes. For DeepSpeech (Hannun et al., 2014) and Kaldi, Khare et al. (Shreya Khare, 2019) proposed a black-box attack based on evolutionary optimization, and also Taori et al. (Taori et al., 2018) present a similar approach in their paper.

Schönherr et al. (Schönherr et al., 2019) published an approach where psychoacoustic modeling, borrowed from the MP3 compression algorithm, was used to re-shape the perturbations of the adversarial examples in such a way as to hide the changes to the original audio below the human hearing thresholds. However, the adversarial examples created in that work need to be fed into the recognizer directly. Concurrently, Abdullah et al. (Abdullah et al., 2019) showed a black-box attack in which psychoacoustics is used to calculate adversarial examples empirically. Their approach focuses on over-the-air attacks, but in many cases, humans can perceive the hidden message once they are alerted to its content. Note that our adversarial examples are conceptually completely different, as we use a target audio file, where we embed the target transcription via backpropagation. The changes, therefore, sound like random noise (see examples available at http://imperio.adversarial-attacks.net). With Abdullah et al.’s approach, an audio file with the spoken target text is taken and changed in a way to be unintelligible for unbiased human listeners, but not for humans aware of the target transcription. This is equally the case for Chen et al.’s (Chen et al., 2020b) recently published black-box attack against several commercial devices, where humans can perceive the target text. As an extension of Carlini’s and Wagner’s attack (Carlini and Wagner, 2018), Qin et al. (Qin et al., 2019) introduced the first implementation of RIR-independent adversarial examples. Unfortunately, their approach only worked in a simulated environment and not for real over-the-air attacks, but the authors also utilize psychoacoustics to limit the perturbations.

In the visual domain, Evtimov et al. (Evtimov et al., 2017) showed one of the first real-world adversarial attacks. They created and printed stickers, which can be used to obfuscate traffic signs. For humans, the stickers are visible. However, they seem very inconspicuous and could possibly fool autonomous cars. Athalye et al. (Athalye et al., 2017) presented another real-world adversarial perturbation on a 3D-printed turtle, which is recognized as a rifle from almost every point of view. The algorithm to create this 3D object not only minimizes the distortion for one image but for all possible projections of a 3D object into a 2D image, hence producing a robust adversarial example.

Recently and independently, Chen et al. (Chen et al., 2020a) showed a first over-the-air attack. Their attack was evaluated against DeepSpeech (Hannun et al., 2014). Note that we focus on generic adversarial examples that work over-the-air for different kinds of rooms. Additionally, we used Kaldi, a hybrid DNN-HMM ASR system that works in a fundamentally different manner than the end-to-end approach of DeepSpeech, which is attacked by Chen et al.

Our approach is the first targeted attack that provides room-independent, robust adversarial examples against a hybrid ASR system. We demonstrate how to generate adversarial examples that are mostly unaffected by the environment, as ascertained by verifying their success in a broad range of room characteristics. We utilize the same psychoacoustics-based approach proposed by Schönherr et al. (Schönherr et al., 2019) to limit the perturbations of the audio signal to remain under, or at least close to, the human thresholds of hearing, and we show that the examples remain robust to playback over the air. The perturbations that remain audible in the adversarial examples that we create, are non-structured noise, so that human listeners cannot perceive any content related to the targeted recognition output. Hence, our attack can be successful in a broad range of possible rooms, without any physical access to the environment (e. g., by playback of inconspicuous media from the Internet), and for which the target recognition output is not at all perceptible by human listeners. It shows the possibility and risk of a new attack vector, as no specialized hardware is needed for the playback and by being insensitive to the rooms in which the attacked systems are being operated.

6. Discussion

Our experiments show that the adversarial examples, which we calculated with the proposed algorithm, remain robust even for high reverberation times or large distances between speaker and microphone. Also, the same adversarial examples can be successfully played over the air, even for setups where no direct line-of-sight exists.

Attack Parameters. Our comparison between the generic and the adapted version of the attack shows that the more powerful generic attack does not only have a similar success rate but can even outperform an adapted version where the attacker has prior knowledge of the target room. Consequently, an attacker only needs to calculate one generic adversarial example to cover a wide range of possible recording setups simultaneously.

For an attack, one successful adversarial example, which remains robust after being replayed (with a WER of [math] %), is already enough. Therefore, the best strategy for an actual attack would be to calculate a set of adversarial examples containing the malicious transcription and to choose the most robust ones. In general, the results indicate a trade-off between the WER and the noise level: if no hearing thresholds are used, the WER is significantly better in comparison to examples with hearing thresholds. Nevertheless, even if the WER is better in cases without hearing thresholds, we have shown that it is indeed possible to calculate over-the-air-robust adversarial examples with hearing thresholds. Those adversarial examples contain less perceptible noise and are, therefore, less likely to be detected by human listeners.

End-to-end ASR systems. End-to-end ASR systems differ widely from the hybrid ASR systems used in this paper. However, the proposed attack only requires the possibility for backpropagation from the output to the input of the recognition network, and can therefore be applied to end-to-end systems. A simulated version of a similar attack with RIRs has been shown by Qin et al. (Qin et al., 2019). An adaptation of this attack is, therefore, most likely transferable to end-to-end ASR systems in the real world.

Black-Box Attack. In a black-box scenario, the attacker has no access to the ASR system. However, even for this more challenging attack, it has been shown that it is possible to calculate adversarial examples, with the caveat that humans can perceive the hidden transcription if they are made aware of it (Chen et al., 2020b). The proposed approach is not easy to apply to black-box adversarial examples like commercial ASR systems such as Amazon’s Alexa. Nevertheless, it should be feasible to use a similar approach in combination with a parameter-stealing attack (Ilyas et al., 2018; Papernot et al., 2016; Tramèr et al., 2016; Papernot et al., 2017; Wang and Gong, 2018). Once the attacker can rebuild their own system, which reassembles the black-box system, the proposed algorithm can be used with that system as well.

Countermeasures. To effectively prevent adversarial attacks, an ASR system needs either some kind of detection mechanism or needs to be hardened against adversarial examples. The detection of adversarial examples for known attacks might be feasible. However, no guarantees can be given against novel attacks in the long term. For this, it is necessary to build the ASR system to be adversarial-example-robust, e. g., by mimicking the human perception of speech similar to images encoded in JPEG format (Bafna et al., 2018). One step in this direction can be to focus the ASR system on only those signal components that are perceptible to the human listener and thus carry semantic information.

Additionally, not only the input data can be utilized to detect adversarial examples, but the ASR system’s DNN can also serve this purpose. To achieve this, the uncertainty of the DNN estimation can be utilized to predict the reliability of the DNN output (Gal and Ghahramani, 2016; Louizos and Welling, 2016; Lakshminarayanan et al., 2017; Däubener et al., 2020). Due to the difficulty to creating robust adversarial example defenses, Carlini et al. proposed a guideline for the evaluation of adversarial robustness, which lists all important properties of a successful countermeasure against adversarial examples (Carlini et al., 2019).

7. Conclusion

In this paper, we have demonstrated that ASR systems are vulnerable against adversarial examples played over the air and we have introduced an algorithm for the calculation of robust adversarial examples. By simulating varying room setups, we can create highly robust adversarial examples that remain successful over the air in many environments.

To substantiate our claims, we performed over-the-air attacks against Kaldi; a state-of-the-art hybrid recognition framework that is used in Amazon’s Alexa and other commercial ASR systems. We presented the results of empirical attacks for different room configurations. Our algorithm can be used with and without psychoacoustic hearing thresholds, limiting the perturbations to being less perceptible by humans. Furthermore, we have shown that it is possible to create targeted robust adversarial examples for varying rooms even if no direct line-of-sight between the microphone and the speakers exists, and even if the test room characteristics are completely unknown during the creation of the example.

Future work should investigate possible countermeasures such as using only the perceptible parts of the audio signal for recognition or using internal statistical information of the hybrid recognizer for detecting attacks.

Acknowledgments

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy - EXC 2092 CaSa - 390781972.

Appendix A Room Layout Plans