Robust End-to-End Speaker Verification Using EEG

TL;DR

This paper shows that combining EEG signals with speech features enhances the robustness of speaker verification systems, especially in noisy environments, using a state-of-the-art deep learning approach.

Contribution

It introduces a novel end-to-end deep learning method that incorporates EEG signals to improve speaker verification performance in noisy conditions.

Findings

EEG signals improve speaker verification accuracy in noisy environments

Concatenating EEG with speech features enhances robustness

EEG-only features can be effective for speaker verification

Abstract

In this paper we demonstrate that performance of a speaker verification system can be improved by concatenating electroencephalography (EEG) signal features with speech signal features or only using EEG signal features. We use state-of-the-art end-to-end deep learning model for performing speaker verification and we demonstrate our results for noisy speech. Our results indicate that EEG signals can improve the robustness of speaker verification systems, especially in noiser environment.

Tables

Table 1. TABLE I: EER on test set for Data set A (40 dB) using LSTM based model

Number of Sentences	MFCC (EER %)	MFCC+EEG (EER %)	EEG (EER %)
3	10	7	3
5	12	10	5
7	14	11	6
10	17	13	8
20	19	14	9
30	22	16	11

Table 2. TABLE II: EER on test set for Data set A (40 dB) using GRU based model

Number of Sentences	MFCC (EER %)	MFCC+EEG (EER %)	EEG (EER %)
3	8	5	3
5	11	8	4
7	14	10	6
10	15	11	7
20	17	12	8
30	19	14	10

Table 3. TABLE III: EER on test set for Data set B (65 dB) using LSTM based model

Number of Sentences	MFCC (EER %)	MFCC+EEG (EER %)	EEG (EER %)
3	12	10	4
5	15	12	6
7	16	13	7
10	20	15	8
20	23	18	9
30	26	19	12

Table 4. TABLE IV: EER on test set for Data set B (65 dB) using GRU based model

Number of Sentences	MFCC (EER %)	MFCC+EEG (EER %)	EEG (EER %)
3	11	9	4
5	14	10	5
7	15	11	7
10	18	16	7
20	21	15	8
30	24	17	11

Full text

I Introduction

Speaker verification is the process of verifying whether an utterance belongs to a specific speaker, based on that speaker’s known utterances. Speaker verification systems are used as authentication system in many voice activated technologies, for example in applications like voice match for Google home. In our work we focused on text independent speaker verification [1, 2]. Deep learning based speaker verification systems [3, 4, 5, 6, 7, 8] are getting popular this days and such systems have improved the performance of speaker verification systems.

Even though deep learning models have improved the state-of-the-art performance of speaker verification systems, their performance is degraded in presence of background noise and they are prone to attacks in the form of voice mimicking by the adversaries who are interested in breaching the authentication system.

We propose to use electroencephalography (EEG) signals to address these challenges. EEG is a non-invasive way of measuring electrical activity of human brain. In [9] we demonstrated that EEG features can help automatic speech recognition (ASR) systems to overcome the performance loss in presence of background noise. Further, prior works explained in the references [10, 11, 12, 13, 14, 15, 16] show that EEG pattern in every individual is unique and it can be used for biometric identification. Motivated by these prior results, we used EEG features to improve the performance of speaker verification systems operating, compared with using acoustic features under two different levels of noisy conditions. The use of EEG features to improve the robustness of speaker verification system is also motivated by the unique robustness to environmental artifacts exhibited by the human auditory cortex [17, 18]. Speaker verification using EEG will also help people with speaking disabilities like broken speech to use voice authentication technologies with better accuracy, thereby improving technology accessibility.

For this work we used the end-to-end speaker verification model explained in [4] as it is the current state-of-the-art model for speaker verification. Major contribution of this paper is the demonstration of improving robustness of speaker verification systems using EEG features.

The rest of this paper is structured as follows. Section II introduces the speaker verification model used in our work, while Section III, IV and V describe details of the EEG database, feature extraction and preprocessing method for the data set. Experimental results and analysis are presented in Section VI, and finally, we conclude the paper in Section VII.

II Speaker Verification Model

We used generalized end-to-end model introduced in [4] for performing text independent speaker verification. We used Google’s Tensorflow deep learning library for implementing the model.

The model consists of a single layer of Long short term memory (LSTM) [19] or Gated recurrent unit (GRU) [20] cell with 128 hidden units followed by a dense layer which is a fully connected linear layer followed by a softmax activation. During training phase, an utterance $i$ of an enrollment candidate $X$ is passed to the LSTM or GRU cell and L2 normalization is applied to the dense layer output to derive the embedding or d vector for that utterance $i$. Similarly d vectors are derived for all the utterances of the enrolment candidate $X$. Since different utterances can have different lengths, we used dynamic recurrent neural network (RNN) cell of tensorflow. The speaker model or centroid for $X$ is defined as the average value of all the d vectors obtained for all the enrollment utterances of $X$.

Consider a set of enrollment candidates {$X_{1}$,$X_{2}$, $\cdots\cdots$, $X_{n}$} and let us assume each $X_{i}$ has $t$ number of utterances. Now as per our earlier definition we can build a set of centroids as {$C_{1}$, $C_{2}$, $\cdots\cdots$, $C_{n}$} where $C_{i}$ is the centroid for $X_{i}$. Now a two dimensional similarity matrix with $n\times t$ rows and $n$ columns is constructed, that computes the cosine similarity between all $C_{i}$’s and d vectors corresponding to utterances from all $X_{i}$’s. The cosine similarity contains learnable parameters that act as scaling factors as explained in [4]. A softmax layer is applied on the similarity matrix. We used the generalized end-to-end softmax loss explained in [4] as the loss function for the model. For any ($C_{i}$, $d_{i}$) pair, where $d_{i}$ is a d vector corresponding to an utterance from any $X_{i}$, the final softmax layer in the model outputs 1 if both $C_{i}$ and $d_{i}$ are derived from the same $X_{i}$, otherwise it outputs 0. Batch size was set to one and gradient descent optimizer Adam [21] was used.

During test time, along with the enrollment utterances from the test set, evaluation utterances are also passed to the trained model. The similarity is calculated between the centroids derived from the enrollment utterances from test set and d vectors corresponding to evaluation utterances. Figure 2 shows the training methodology and Figure 3 shows the testing methodology of the verification model. Figure 2 is adapted from [4]. By the term utterance through out this paper, we refer to the Mel-frequency cepstral coefficients (MFCC 13) or EEG features or concatenation of MFCC 13 and EEG features, depending on how the speaker verification model was trained.

III Design of Experiments for building the database

For this work, we have built two database of synchronized speech-EEG signals. For the data set A, 10 subjects from UT Austin (5 of them are female) took part in the experiment. While for the data set B, 8 subjects from UT Austin (3 of them are female) took part in the experiment. Since this work is about speaker verification, we don’t require all the subjects to be native speakers. In our case, two of them are Chinese native speakers. And all subjects were graduate, undergraduate students in their early twenties.

For the both data sets, the subjects were asked to speaker the first 30 sentences from the USC-TIMIT database [22] and their synchronized speech-EEG signals were recorded. The process was repeated for two more times. For the data set A, we had 30 speech-EEG recording samples for each sentence, and the level of background noise is 40 dB, generated by lab air conditioner fan. For the data set B, we had 24 speech-EEG recording samples for each sentence, and the level of background noise is 65 dB, generated by music of lab computer.

The Brain Vision EEG recording hardware is used in the experiment. The Figure 1 shows our EEG cap, from the figure, we can find that it contains 32 wet EEG electrodes with one electrode as ground included. The EEG sensor topological location mapping is obtained from EEGLab[23], which is based on standard 10-20 EEG sensor placement method for 32 electrodes.

During the model training phase, for data set A, speech-EEG data of 8 subjects is used, with the remaining 2 subjects’ data as test set. While for data set B, speech-EEG data of 6 subjects is used for training, with remaining 2 subjects’ data as test set.

IV EEG and Speech feature extraction details

For preprocessing of EEG we followed the same method as described by the authors in [24, 9]. EEG signals were sampled at 1000Hz and a fourth order IIR band pass filter with cut off frequencies 0.1Hz and 70Hz was applied. A notch filter with cut off frequency 60 Hz was used to remove the power line noise. EEGlab’s [23] Independent component analysis (ICA) toolbox was used to remove other biological signal artifacts like electrocardiography (ECG), electromyography (EMG), electrooculography (EOG) etc from the EEG signals. We extracted five features the same as the ones used by the authors in [24, 9], namely root mean square, zero crossing rate, moving window average, kurtosis and power spectral entropy with frequency bands value equal to none (power per band was same as the power spectral density). All EEG features were extracted at a sampling frequency of 100 Hz. For this feature set, EEG feature dimension was 31(channels) $\times$ 5 or 155.

The recorded speech signal was sampled at 16KHz frequency. Mel-frequency cepstrum (MFCC) was extracted as features for speech signal. We extracted MFCC 13 features. In order to avoid seq2seq problem, the sampling frequency of MFCC features were also fixed at 100 Hz the same as it of EEG features.

V EEG Feature Dimension Reduction Algorithm Details

We used non linear dimension reduction methods to denoise the EEG feature space. The tool we used for this purpose was Kernel Principle Component Analysis (KPCA)[25]. We plotted cumulative explained variance versus number of components to identify the right feature dimension as shown in Figure 4. We used KPCA with polynomial kernel of degree 3[24, 9]. The cumulative explained variance plot is not supported by the library for KPCA as KPCA projects features into different feature space, hence for getting explained variance plot we used normal PCA but after identifying the right dimension we used KPCA to perform dimension reductions.

VI Results

As explained earlier, for data set A we used data from first 8 subjects or 720 sentence utterances as training set, last two subjects or 180 sentence utterances as testing set. For data set B we used data from first 6 subjects or 540 sentence utterances as training set, last two subjects or 180 sentence utterances as testing set. There are 90 utterances per each subject.

Equal error rate (EER) defined in [3] was used as the evaluation metric to evaluate the model. Table I shows the obtained results on the less noisy data set A when we used LSTM as the RNN cell in the verification model and results clearly indicate that combining EEG features with acoustic features help in reducing the EER. In the table results are shown for number of sentence = {3, 5, 7, 10, 15, 20, 30}. Number of sentence = 3 implies during training, for first training step, features (MFCC or MFCC+EEG or EEG) corresponding to first 3 sentence utterances are selected randomly from any two subjects from the training set and during second training step, features corresponding to next 3 sentence utterances are selected randomly from any two subjects and so on till the 90 sentence utterance is selected from the training set. So for this example, 30 training steps constitute one epoch. The model is trained for sufficient number of epochs to make sure it sees all the utterances present in the training set. In every epoch any two subjects are randomly chosen.

During test time we have data from two subjects and number of sentences equal to 3 implies, for the first testing step features corresponding to first 3 sentence utterances from subject 1 and subject 2 are selected as enrollment utterances and features corresponding to next 3 sentence utterances from subject 1 and subject 2 are selected as evaluation utterances and EER is calculated. For second testing step, the features used as evaluation utterances in first step will become new enrollment utterances and features corresponding to next 3 sentence utterances from subjects 1, 2 will become new evaluation utterance and new EER is calculated. So for this example,there will be a total of 30 testing steps. EER value of 10%, 7% or 3% for number of sentences equal to 3 as seen from the Table I, corresponds to the average of all the 30 testing step EER values. Similarly model was trained and tested for number of sentences = {3, 5, 7, 10, 15, 20, 30}. As 20 is not a factor of 90, for number of sentences equal to 20, there are five training steps in one epoch, five testing steps but the last step will contain only the features corresponding to last 10 sentence utterances. Figure 5 shows the training loss convergence for the LSTM model for number of sentences equal to 3 on data set A when trained using EEG features.

Table II shows the test time result using GRU model for Data Set A, Table III shows the test time result using LSTM model for Data Set B and Table IV shows the test time result using GRU model for Data Set B. The results shows that GRU model is almost better than LSTM model in all cases. The reason is that GRU is less complex than LSTM, and more suitable for a small size data set like our data set.

As can be seen from the tables, the EER of only using MFCC features is more prone to be worse under increasing level of environment noise compared with using MFCC+EEG features or only using EEG features, which indicates that EEG features are more robust for speaker verification.

VII Conclusions

In this paper we demonstrated the feasibility of using EEG signals to improve the robustness of speaker verification systems operating in very noisy environment. We demonstrated that for the two levels of noisy data, 40 dB and 65 dB, combination of MFCC and EEG, or only EEG features always resulted in lower EER.

Our overall results indicate that EEG features are less affected by background noise compared with MFCC features, which indicates that they are helpful in improving robustness of verification systems operating in presence of high background noise.

We further plan to publish our speech EEG data base used in this work to help advancement of the research in this area. For future work, we will work on building a much larger speech EEG data base and validate the results for larger data base.

VIII Acknowledgements

We would like to thank Kerry Loader from Dell, Austin, TX for donating us the GPU to train the models used in this work.