Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spotting

TL;DR

This paper introduces a max-pooling loss function for training LSTM networks aimed at small-footprint keyword spotting, demonstrating improved performance over traditional methods with low resource requirements.

Contribution

The study proposes a novel max-pooling loss training approach for LSTM-based keyword spotting, showing it outperforms standard cross-entropy training and baseline models.

Findings

Max-pooling loss trained LSTM outperforms cross-entropy trained LSTM.

Max-pooling loss with pre-training yields best performance.

Achieves 67.6% relative reduction in AUC over baseline DNN.

Abstract

We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothing based evaluation approach is employed to measure keyword spotting performance. Our experimental results show that LSTM models trained using cross-entropy loss or max-pooling loss outperform a cross-entropy loss trained baseline feed-forward Deep Neural Network (DNN). In addition, max-pooling loss trained LSTM with randomly initialized network performs better compared to cross-entropy loss trained LSTM. Finally, the max-pooling loss trained LSTM initialized with a cross-entropy pre-trained network shows the best performance, which yields $67.6\%$ relative reduction compared to baseline feed-forward DNN in Area Under the Curve (AUC) measure.

Tables

Table 1. Table 1 : Relative change of AUC for LSTM models compared to the baseline feed-forward DNN. Lower AUC indicates better performance.

model	DNN	LSTM
loss function	xent	xent	maxpool
			random init	xent pretrain
AUC change	$0\%$	$-34.4\%$	$-48.2\%$	$-67.6\%$

Equations

$${\mathbf{i}}_t=\sigma ({\mathbf{W}}_{ix}{\mathbf{x}}_t+{\mathbf{W}}_{ir}{\mathbf{r}}_{t-1}+{\mathbf{W}}_{ic}{\mathbf{c}}_{t-1}+{\mathbf{b}}_i)$$

$${\mathbf{f}}_t=\sigma ({\mathbf{W}}_{fx}{\mathbf{x}}_t+{\mathbf{W}}_{fr}{\mathbf{r}}_{t-1}+{\mathbf{W}}_{fc}{\mathbf{c}}_{t-1}+{\mathbf{b}}_f)$$

$${\mathbf{c}}_t={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot g({\mathbf{W}}_{cx}{\mathbf{x}}_t+{\mathbf{W}}_{cr}{\mathbf{r}}_{t-1}+{\mathbf{b}}_c)$$

$${\mathbf{o}}_t=\sigma ({\mathbf{W}}_{ox}{\mathbf{x}}_t+{\mathbf{W}}_{or}{\mathbf{r}}_{t-1}+{\mathbf{W}}_{oc}{\mathbf{c}}_t+{\mathbf{b}}_o)$$

$${\mathbf{m}}_t={\mathbf{o}}_t\odot h({\mathbf{c}}_t)$$

$${\mathbf{r}}_t={\mathbf{W}}_{rm}{\mathbf{m}}_t$$

$${\mathbf{y}}_t=\varphi ({\mathbf{W}}_{yr}{\mathbf{r}}_t+{\mathbf{b}}_y)$$

$$n=n_c\times n_r\times 4+n_i\times n_c\times 4+n_r\times n_o+n_c\times n_r+n_c\times 3$$

$${\mathcal{L}}_t^{xent}=-\sum _{k=1}^K{z}_t^k\ln y_t^k$$

$${\mathcal{L}}_t^{xent}=-\ln y_t^{k_t^{†}}$$

$${\mathcal{L}}_T^{xent}=\sum _{t=1}^T{\mathcal{L}}_t^{xent}=-\sum _{t=1}^T\ln y_t^{k_t^{†}}$$

$${\mathcal{L}}_T^{maxpool}=-\sum _{t\in \hat{\mathbf{L}}}\ln y_t^{k_t^{†}}-\sum _{p=1}^P\ln y_{l_p^{†}}^{k_p^{†}}$$

Full text

**Index Terms— ** LSTM, keyword spotting, max-pooling loss, small-footprint

1 Introduction

Keyword spotting has been an active research area for decades. Different approaches have been proposed to detect the words of interest in speech utterances. As one solution, a general large vocabulary continuous speech recognition (LVCSR) system is applied to decode the audio signal, and keyword searching is conducted in the resulting lattices or confusion networks [1, 2, 3, 4]. These methods require relatively high computational resources for the LVCSR decoding, and also introduce latency.

Small-footprint keyword spotting systems have been increasingly attracting attention. Voice assistant systems such as Alexa on Amazon Echo deploy a keyword spotting system on device, and only stream audio to the cloud for LVCSR when the keyword is detected on device. For such applications, accurate on-device keyword spotting running with low CPU and memory is critical [5]. It needs to run with high recall to make devices easy to use, while having low false accepts to mitigate privacy concerns. Latency has to be low as well. A traditional approach employs Hidden Markov Model (HMM) to model both keyword and background [6, 7, 8]. The background includes non-keyword speech, or non-speech noise etc. This background model is also named filler model in some literatures. It could involve loops over simple speech/non-speech phones, or for more complicated cases, normal phone set or confusing word set. Viterbi decoding is used to search the best path in the decoding graph. The keyword spotting decision can be made based on the likelihood comparison of keyword and background models. Gaussian Mixture Model (GMM) was commonly used in the past to model the observed acoustic features. With DNN becoming mainstream for acoustic modeling, this approach can be extended to include discriminative information by incorporating a hybrid DNN-HMM decoding framework [9].

In recent years, there are keyword spotting systems built on DNN or Convolutional Neural Network (CNN) directly, with no HMM involved in the system [10, 11, 12, 13]. During decoding time, framewise keyword posteriors are smoothed. The system is triggered when smoothed keyword posteriors exceed a pre-defined threshold. The trade off between balancing false rejects and false accepts can be performed by tuning the threshold. Context information is taken care of by stacking frames as input. Some keyword spotting systems are built on Recurrent Neural Network (RNN) directly. Particularly, bidirectional LSTM is used to search for keywords in audio streams when latency is not a hard constraint [14, 15, 16, 17].

We are interested in a small-footprint keyword spotting system that runs on low CPU and memory utilization, with low latency. This low latency constraint makes bidirectional LSTM not a proper fit in principle. Instead, we focus on training a unidirection LSTM model using two different loss functions: cross-entropy loss and max-pooling based loss [18]. Applying the max-pooling loss function to LSTM training for keyword spotting is the main contribution of this paper.

During decoding time, the system is triggered when the keyword posterior smoothed by averaging the output of a sliding window is above a threshold. Considering the practical use case, our keyword spotting system is designed to lock out for some time after each detection, to avoid unnecessary false accepts and reduce decoding computational cost.

The remaining part of this paper is organized as follows: Section 2 describes our LSTM based keyword spotting system, which includes the LSTM model, training loss functions and performance evaluation details. Experimental setup and results are included in Section 3. Section 4 is for conclusion and future work.

2 System Overview

As shown in Figure 1, Log Mel Filter-Bank Energies (LFBEs) are used as input acoustic features for our keyword spotting system. We extract 20 dimensional LFBEs over 25ms frames with a 10ms frame shift. The LSTM model is used to process input LFBEs. Our system has two targets in the output layer: non-keyword and keyword. The output of the keyword spotting system is passed to an evaluation module for decision making.

2.1 LSTM

Different from feed-forward DNN networks, RNNs contain cyclic connections which can be used to model sequential data . This makes RNNs a natural fit to model temporal information within continuous speech frames. However, traditional RNN structures suffer from the vanishing gradient problem, which prevents them from effectively modeling long context in the data. To overcome this, LSTMs contain memory blocks [19, 20]. Each block contains one or more memory cells, as well as input, output and forget gates. These three gates control the information flow within the associated memory block. Sometimes a projection layer is added on top of the LSTM output, to reduce model complexity [21]. A typical LSTM component with projection layer is shown in Figure 2. For the sake of clarity, a single LSTM block is shown here.

Given a sequence of T frames $\mathbf{X}=(\mathbf{x}_{1},\ldots,\mathbf{x}_{T})$, let ${\mathbf{i},\mathbf{o},\mathbf{f},\mathbf{c}}$ denote the input, output, forget gates, and the memory cell, and ${\mathbf{Y}=(\mathbf{y}_{1},\ldots,\mathbf{y}_{T})}$ be the output. The LSTM computes the gate activations and output at time $t$ as follows:

[TABLE]

Here $\mathbf{W}_{\ast}$ matrices label the connection weights. E.g., $\mathbf{W}_{ix}$, $\mathbf{W}_{ir}$ and $\mathbf{W}_{ic}$ represent the weight matrices from the input $\mathbf{x}$, recurrent feedback $\mathbf{r}$ and cell $\mathbf{c}$ respectively. Note that the peephole connections $\mathbf{W}_{ic}$, $\mathbf{W}_{fc}$ and $\mathbf{W}_{oc}$ are diagonal matrices. The $\mathbf{b}_{\ast}$ terms represent the bias vectors for different components of the model. E.g., $\mathbf{b}_{i}$ is the bias for input gate activation.

A projection layer is added to the LSTM output. That is, $\mathbf{W}_{rm}$ linearly maps $\mathbf{m}_{t}$ to a lower dimensional representation $\mathbf{r}_{t}$, which is the recurrent signal. The network output $\mathbf{y}_{t}$ is computed based on the projection layer output $\mathbf{r}_{t}$ as well.

Regarding the activation functions, we use logistic sigmoid function as ${\sigma()}$ for gate activations, $\tanh$ as $g()$ and $h()$ for cell input and output, and softmax as $\phi()$ for output layer. $\odot$ is the element-wise product of vectors.

Finally, the complexity of the model described above can be calculated as

[TABLE]

where $n_{c}$ is the number of memory cells (we only consider the case of single memory cell per block, thus here $n_{c}$ is also the number of memory blocks), $n_{r}$ is the dimension of projection layer, ${n_{i}}$ and ${n_{o}}$ denote the dimension of input and output respectively.

2.2 Loss functions

For our experiments, we consider two different types of loss functions: cross-entropy loss and max-pooling loss.

2.2.1 Cross-entropy

Cross-entropy (xent) has been widely applied as a loss function for DNN and RNN training [22]. Let $K$ be the total number of classes. Given a sequence of $T$ frames $\mathbf{X}=(\mathbf{x}_{1},\ldots,\mathbf{x}_{T})$, where $\mathbf{x}_{t}$ is the feature vector of the $t$th frame, let $\mathbf{y}_{t}=(y_{t}^{1},\ldots,y_{t}^{K})$ denote the $K$-dimensional output of the network for $\mathbf{x}_{t}$, and let $\mathbf{z}_{t}=(z_{t}^{1},\ldots,z_{t}^{K})$ denote the corresponding target vector. The cross-entropy loss for the $t$th frame is calculated as follows:

[TABLE]

The $1$-of-$K$ coding is usually used for target vector $\mathbf{z}_{t}$. That is, if the $t$th frame vector $\mathbf{x}_{t}$ is aligned with class $k$, the $K$-dimensional vector $\mathbf{z}_{t}$ has value $1$ for the $k$th element, with all other elements being [math]. Let $k^{\dagger}_{t}$ denote the aligned class for the $t$th frame. The cross-entropy loss for the $t$th frame can be formulated as

[TABLE]

Then the cross-entropy loss for the whole $T$ frame sequence is:

[TABLE]

2.2.2 Max-pooling

We propose to train the LSTM for keyword spotting using a max-pooling based loss function. Given that the LSTM has the ability to model long context information, we hypothesize that there is no need to teach the LSTM to fire every frame within the keyword segment. Instead, we want to teach the LSTM to fire at its highest confidence time. The LSTM should fire near the end of keyword segment in general, where it has seen enough context to make a decision. A simple way is to back-propagate loss only from the last frame or last several frames for updating the weights. But our initial experiments indicate that the LSTM does not learn much from this scheme. Hence we employ a max-pooling based loss function to let the LSTM pick the most informative keyword frames to teach itself. This also helps mitigate issues potentially caused by inaccurate frame alignment around keyword segment boundaries. Max-pooling loss can be viewed as a transition from frame-level loss to segment-level loss for keyword spotting model training.

Alternative segment-level loss functions include different statistics of frame-level keyword posteriors within a keyword segment, e.g., the geometric mean etc. There have been literatures on training LSTMs using Connectionist Temporal Classification (CTC) [14, 15, 16, 23] for keyword spotting tasks as well. In addition, architectures that combine LSTMs and CNNs have been applied to different tasks [24, 25]. Typically LSTM is added on top of CNN layers, where CNN layers with pooling are used to extracted features as LSTM input, and LSTM output is used for prediction.

Let $Q$ denote the cardinality of target keyword set. When we consider word level labels, there are in total $Q+1$ classes ($K=Q+1$), with one additional class used to label those frames aligned with background. For the $T$ input frames $\mathbf{X}=(\mathbf{x}_{1},\ldots,\mathbf{x}_{T})$, if there are $P$ keywords instances inside, we use $\mathbf{l}_{p}$ to denote a continuous frame index range whose frames are aligned with the $p$th keyword. As a result, the $K$-dimensional target vector $\mathbf{z}_{p}$ is the same for all frames within $\mathbf{l}_{p}$.

Let $\mathbf{L}=(\mathbf{l}_{1},\ldots,\mathbf{l}_{P})$ represent a collection of frame index ranges for all $P$ keywords instances in $\mathbf{X}$, and $\hat{\mathbf{L}}$ be a collection of all the indices for the remaining frames which are not aligned to any keyword (i.e. background frame indices). We use $k^{\dagger}_{p}$ to represent the target label for frames inside $\mathbf{l}_{p}$, and $l^{\dagger}_{p}$ to label the specific one frame within $\mathbf{l}_{p}$ whose posterior for $k^{\dagger}_{p}$ is the maximum. The max-pooling loss proposed for the input sequence $\mathbf{X}$ can be calculated as

[TABLE]

The first item states that we calculate the cross-entropy loss for input frames not aligned to any keyword. The second item shows how we do max-pooling for keyword aligned frames. In more details, for the frames of the $p$th segment (index range $\mathbf{l}_{p}$), they are aligned to keyword $k^{\dagger}_{p}$. We only back propagate for a single frame (index $l^{\dagger}_{p}$) whose posterior for target $k^{\dagger}_{p}$ is the largest among all frames within current segment $\mathbf{l}_{p}$, and discard all other frames within current segment.

The idea of max-pooling loss is shown in Figure 3, where filled frames are aligned with the keywords, and empty frames are for background. Given an input sequence of frames, within each keyword segment, only the frame which has the maximum posterior for corresponding keyword target is kept, while all other frames within the same keyword segment are discarded. All background frames are kept.

We consider two cases for max-pooling loss based LSTM training: one starts with a randomly initialized model, and the other uses a cross-entropy loss pre-trained model. With a randomly initialized model, max-pooling loss based LSTM training may not learn well in the first few epochs with rather random keyword firing. The idea is to take the advantages of both cross-entropy and max-pooling loss training. With a cross-entropy trained LSTM as the initial model to start max-pooling training, it already learns some basic knowledge about target keywords. This could provide a better initialization point, and faster convergence to a better local optimum.

2.3 Evaluation Method

We consider a posteriors smoothing based evaluation scheme. To detect the keyword, given input audio, the system computes smoothed posteriors based on a sliding context window containing $N_{ctx}$ frames. When the smoothed posterior for the keyword exceeds a pre-defined threshold, this is considered as a firing spike. The system is designed to shut down for the following $N_{lck}$ frames. This lockout period of length $N_{lck}$ is for the purpose of reducing unnecessarily duplicated detections during the same keyword segment, as well as reducing decoding computational cost.

For our use case, we allow a short latency period with ${N_{lat}}$ frames after each keyword segment. That is, if the system fires within the ${N_{lat}}$-frame window right after a keyword segment, we still consider the firing as being aligned with the corresponding keyword. This latency window does not introduce significant delay in perception, and it could mitigate the possible issues of inaccurate keyword alignment boundaries in evaluation.

Finally, the first firing spike within each keyword segment plus latency window is considered as a valid detection. Any other firing spikes within the same keyword segment plus latency window, or outside any keyword segment plus latency window, are counted as false accepts. Two metrics are used to measure the system performance: miss rate, which is one minus recall, and false accept rate, which is a normalized value of false accepts.

Figure 4 illustrates the idea of our evaluation approach. As examples, there are two input audio streams. The keyword segment length varies depending on the way the keyword is spoken. Each keyword segment is followed by a fixed length latency window. The keyword segments are labeled by blocks with vertical line fill, while the follow-on latency windows are labeled by blocks with horizontal line fill. There is a system lock out period by design after each firing spike. For the first audio, there are two false accepts (FAs) with system firing in the region outside any keyword segment plus latency window. The true accepts (TAs) happen as the first detection in each keyword segment plus latency window. True accepts could happen either in the keyword segment, or in the following latency window. For the second audio, false accepts happen as additional firing spikes within the same keyword segment plus latency window which already has a true accept.

For our system, we use $30$ frames ($N_{ctx}=30$) for posteriors smoothing, $40$ frames ($N_{lck}=40$) as the lockout period, and $20$ frames ($N_{lat}=20$) as the allowed latency window length after each aligned wake word segment.

3 Experimental Results

For our experiments, the word ’Alexa’ is chosen as the keyword. We use an in-house far-field corpus which contains far-field data collected under different conditions. This dataset contains an order of magnitude more instances of keyword and background speech utterances than the largest previous studies [10, 12] for both training and testing. Our data is collected in a far-field environment, which is a more challenging task by nature. Considering the large size of our corpus, the development set partition is sufficient to tune parameters, and the test set partition is large enough to show strong statistical difference.

Since we only target for one keyword ’Alexa’, a binary target set is used for our experiments. Frames of background have target label [math], while frames aligned with keyword have target $1$. We train a feed-forward DNN model as the baseline based on the model structure and training described in [10], with some adaptations to our experimental setup and use case. We compare it with the LSTM models trained with cross-entropy loss and max-pooling loss.

3.1 Model Training

The GPU-based distributed trainer described in [26] is used for our experiments. A performance based learning rate schedule is used for our model training. To elaborate, for each training epoch, if the loss on the dev set degrades compared to the previous epoch, the learning rate is halved, and current epoch is repeated with reduced learning rate. Training process terminates when either the minimal learning rate (for our case, a factor of $0.5^{8}$ of initial learning rate), or the maximum number of epochs is reached (we limit our training to be $20$ epochs). The initial learning rate and batch size are tuned on the development set.

The baseline feed-forward DNN has four hidden layers, with 128 nodes per hidden layer. Sigmoid function is used as activation. A stack of 20 frames on the left and 10 frames on the right are used to form an input feature vector. Note that the right context cannot be too large, since it introduces latency. There are in total $\sim 129K$ parameters with the DNN model. Layerwise pre-training is used for the DNN. Initial learning rate for DNN training is $0.0005$, and batch size is $256$.

For LSTM training with different loss functions, we use a single layer of unidirectional LSTM with 64 memory blocks and a projection layer of dimension 32. This serves the purpose of low CPU and memory, as well as low latency. For input context, we consider $10$ frames on the left and $10$ frames on the right. Note that we still use $10$ frames as left context for LSTM input, though the LSTM learns past frames’ information by definition. By doing this our DNN and LSTM training setup are aligned better for comparison, and past information is further imposed for LSTM training. Our LSTM has $\sim 118K$ parameters. For random initialization, the LSTM parameters are initialized with a uniform distribution $U[-0.2,0.2]$ for weights, and constant $0.1$ for bias. The initial learning rates are chosen to be $0.00001$, $0.00005$ and $0.00005$ for the cases of cross-entropy loss, max-pooling loss with randomly initialized model, and max-pooling loss initialized with a cross-entropy pre-trained model.

3.2 System Performance

We use the evaluation approach described in Section 2.3 on our test dataset. The performance of the DNN and LSTM models are shown in Figure 5.

We plot detection (DET) curves in a low miss rate range, i.e., $\leq 20\%$ for this case. Here the false accept rate is computed by normalizing the false accept counts with the total number of test data utterances. The x-axis labels false accept rate, and the y-axis labels miss rate. Lower numbers indicate better performance. The blue solid curve represents the baseline feed-forward DNN trained using cross-entropy loss. The LSTM models trained using cross-entropy loss, max-pooling loss with random initialization, and max-pooling loss with cross-entropy pre-training, are labeled by the green dashed, red dash-dot and cyan dotted curves respectively. Absolute numbers of false accepts have been obscured in this paper due to confidentiality reasons. Instead, we plot false accept rates up to a multiplicative constant. The false accept range considered in our experiments is aligned with a low value range which can be considered for production deployment purpose.

In the selected low miss rate range, LSTM models outperform the baseline feed-forward DNN. In terms of different loss functions for LSTM training, max-pooling loss with random initialization is superior to cross-entropy loss. LSTM trained using max-pooling loss with cross-entropy loss pre-training yields the best results. We compute the Area Under the Curve (AUC) numbers for quantitative comparison of different models. AUC is computed on DET curves and hence lower is better. The relative changes of AUC for LSTM models compared to the baseline DNN are summarized in Table 1. Our experimental results indicate that in the $\leq 20\%$ low miss rate range, compared to a cross-entropy loss trained baseline DNN, cross-entropy loss trained LSTM results in $34.4\%$ relative reduction in AUC. The LSTM model trained using max-pooling loss with random initialization further shows $48.2\%$ relative reduction in AUC. The best performance comes from the LSTM trained using max-pooling loss with cross-entropy pre-training, which yields $67.6\%$ AUC reduction compared to the baseline DNN.

4 Conclusion and Future Work

We present our work of training a small-footprint LSTM to spot the keyword ’Alexa’ in far-field conditions. Two loss functions are employed for LSTM training: one is cross-entropy loss, and the other is max-pooling loss proposed in this paper. A smoothed posterior thresholding approach is used for evaluation. Keyword spotting performance is measured using miss rate and false accept rate. We show that LSTM performs better than DNN in general. The best LSTM system, which is trained using max-pooling loss with cross-entropy loss pre-training, reduces the AUC number by $67.6\%$ in the low miss rate range.

For future work, we plan to add weighting to max-pooling loss based LSTM training, i.e., scale the back-propagated loss for the selected keyword frames. It is of interest to see if LSTM performance can be further improved by varying model structures, e.g., adding additional feed-forward layers on top of the LSTM component. We also plan to benchmark max-pooling loss performance against other segmental level loss functions, e.g., geometric mean of framewise keyword posteriors within each keyword segment, CTC etc, for our keyword spotting experiments.