Toeplitz Inverse Covariance based Robust Speaker Clustering for Naturalistic Audio Streams

TL;DR

This paper introduces a robust speaker clustering method for diarization in naturalistic audio streams using Toeplitz Inverse Covariance matrices to model sequential i-vector correlations, significantly reducing diarization error rates.

Contribution

It presents a novel model-based clustering approach leveraging Toeplitz Inverse Covariance matrices and an EM algorithm variant for improved speaker diarization accuracy.

Findings

43.22% reduction in DER on CRSS-PLTL corpus

29.37% reduction in DER on AMI subset IS1000a

9.21% reduction in DER on AMI subset IS1003b

Abstract

Speaker diarization determines who spoke and when? in an audio stream. In this study, we propose a model-based approach for robust speaker clustering using i-vectors. The ivectors extracted from different segments of same speaker are correlated. We model this correlation with a Markov Random Field (MRF) network. Leveraging the advancements in MRF modeling, we used Toeplitz Inverse Covariance (TIC) matrix to represent the MRF correlation network for each speaker. This approaches captures the sequential structure of i-vectors (or equivalent speaker turns) belonging to same speaker in an audio stream. A variant of standard Expectation Maximization (EM) algorithm is adopted for deriving closed-form solution using dynamic programming (DP) and the alternating direction method of multiplier (ADMM). Our diarization system has four steps: (1) ground-truth segmentation; (2) i-vector extraction; (3) post-processing (mean subtraction, principal component analysis, and length-normalization) ; and (4) proposed speaker clustering. We employ cosine K-means and movMF speaker clustering as baseline approaches. Our evaluation data is derived from: (i) CRSS-PLTL corpus, and (ii) two meetings subset of the AMI corpus. Relative reduction in diarization error rate (DER) for CRSS-PLTL corpus is 43.22% using the proposed advancements as compared to baseline. For AMI meetings IS1000a and IS1003b, relative DER reduction is 29.37% and 9.21%, respectively.

Figures

Equations

$$\mathbf{S}={\mathbf{S}}_{\mathbf{ubm}}+\mathbf{Tmat}\cdot \mathbf{w}$$

$$\text{minimize}\sum _{i=1}^K\sum _{X_t\in C_i}N(X_t,{\Theta }_i)+\beta 1_{X_{t-1}\notin C_i}$$

$$\sum _{X_t\in C_i}N(X_t,{\Theta }_i)=-|C_i|(\log \text{det}({\Theta }_i)+\text{tr}(S_i{\Theta }_i))+\gamma$$

$$\begin{gathered} \text{minimize}-\log \text{det}({\Theta }_i)+\text{tr}(S_i{\Theta }_i)+\frac{1}{|C_i|}||\lambda \circ {\Theta }_i|{|}_1 \\ \text{subject to }{\Theta }_i\in \mathbb{T} \end{gathered}$$

$$\text{DER}=\frac{{\alpha }_{fa}+{\alpha }_{miss}+{\alpha }_{err}}{{\alpha }_{total}}$$

Full text

Index Terms: Expectation maximization, i-Vector, Markov random field, Peer-led team learning, Speaker clustering, Toeplitz Inverse Covariance.

1 Introduction

Speaker diarization answers "who spoke and when?" in a multi-speaker audio stream [1]. Some of the practical applications of diarization technology include information retrieval [2], broadcast news, meeting conversations, telephone calls, VoIP, digital audio logging [3] and interaction analysis in Peer-Led Team Learning (PLTL) groups [4, 5, 6, 7]. Diarization is a challenging task for naturalistic audio streams as they contain short conversational turns, overlapped speech, noise and reverberation [8, 9]. Variable length speech segments and unsupervised clustering step are two main challenges from modeling perspective. NIST Rich Transcription (RT) evaluations involved broadcast news data and meeting recordings for diarization study [1]. Summed-channel telephone speech from NIST SRE evaluations have been used in diarization studies [4].

Most diarization systems have following five components: (i) speech activity detection (SAD) - to remove non-speech from audio [10, 11]; (ii) speaker change detection/segmentation; (iii) speaker modeling [12, 13]; (iv) speaker clustering; and (v) re-segmentation - that re-aligns each speech frame using speaker models obtained from the clustering module.

Speaker clustering is the most important component in diarization pipeline due to which, several speaker clustering approaches have been studied. These include, but not limited to, bottom-up agglomerative hierarchical clustering (AHC) [14], Cosine K-means [15], Cosine mean-shift [16], top-down approach [17], integer linear programming based method [18], Viterbi-based models [19], PLDA i-vector scoring [20], mixtures of von Mises-Fisher Distributions [21] etc. Recent advancements in speaker clustering still remain inadequate for naturalistic audio streams such as CRSS-PLTL corpus [6] as those do not exploit sequential correlation in speaker turns belonging to same speaker. In this study, we explore the correlation between different speech segments (i-vectors) belonging to same speaker for diarization task. We formulate speaker clustering as a structured inverse covariance estimation problem known as Toeplitz graphical lasso. Our work is motivated by recent success of such approaches in unsupervised analysis of physiological signals and real-world driving data [22]. We perform comparison of proposed Toeplitz Inverse Covariance (TIC) speaker clustering with cosine K-means and movMF baseline methods [21]. In this study, we employ i-vector features for modeling the speakers. Naturalistic audio data from CRSS-PLTL corpus is used in our experiments [23]. PLTL is a STEM education model where a peer-leader facilitate problem-solving among 6-8 students in weekly sessions [24]. We also choose two meetings subset of AMI corpus for comparative evaluations [25].

2 i-Vector Speaker Model

Diarization involve extracting i-Vectors from short speech-segments (typically 1s) unlike speaker verification where complete utterance is used. Numerous techniques exist for clustering i-Vectors using cosine distance [16]. The i-vector framework combines the speaker and channel variability sub-spaces of linear distortion model into a total-variability space represented by matrix $\mathbf{Tmat}$ [13, 26]. A speaker and channel dependent Gaussian Mixture Model (GMM) supervector, $\mathbf{S}$ is decomposed as

[TABLE]

where $\mathbf{S_{ubm}}$ is the Universal Background Model (UBM) supervector [13]. In Eqn. 1, the latent variables, $\mathbf{w}$ are low-rank factors known as i-vectors. The total-variability matrix, $\mathbf{Tmat}$ is a low-rank projection matrix that maps high-dimensional speaker supervector to a low-dimensional total-variability space [13, 26]. $\mathbf{Tmat}$ is estimated using standard expectation maximization (EM) algorithm on background speech data [27].

3 Proposed Speaker Clustering

Toeplitz Inverse Covariance (TIC)-based clustering was found to be suitable for segmenting the real-world time-series data such as fitness-tracker and driving data [22]. Such temporal data has complicated structure where the underlying sequences of few fixed states repeat in definitive patterns. Robust speaker clustering task possess a similar property, i.e., a small set of speakers (such as 10 for PLTL) repeat throughout the audio stream taking different conversational turns. In this section, we describe the TIC based speaker clustering followed by discussion of experimental results in next section.

In the proposed approach, each unique speaker is represented as a correlation network modeled using MRF. Such MRF network capture the interrelationship between different i-vectors (segment-level) belonging to corresponding (one) speaker. Thus, each speaker is modeled by their corresponding MRF. A variant of standard expectation maximization (EM) algorithm is leveraged for estimating the MRF model for each speaker. First of all, the speaker models are initialized. Next, EM iterations run alternately between Expectation (E-step) and Maximization (M-step). During the E-step, feature vectors are assigned to speaker clusters. Next in the M-step, model parameters are updated using dynamic programming (DP) and alternating direction method of multipliers (ADMM) [22]. This process of E-step followed by M-step is repeated until convergence. Essentially, the proposed TIC approach views the temporal order of i-vectors as a sequence of states characterized by their MRF correlation network. Unlike centroid-based approaches such as cosine K-means, proposed approach models the inner structure present in the i-vector feature space. This leads to enhanced clustering performance. Each MRF models the time-invariant partial correlation structure in all feature vectors belonging to the corresponding speaker [22].

3.1 E-step: Assign feature vectors to clusters (Algorithm 1)

In this step, we fix the current value of cluster (model) parameter i.e. inverse covariance matrix, $\mathbf{\Theta}$ and solve the following combinatorial problem for obtaining the cluster assignment set, $\mathbb{C}=\{C_{1},C_{2},\cdots,C_{K}\}$ given by:

[TABLE]

The Eqn. 2 assign each of the $T$ i-vectors to one of the $K$ speaker clusters by jointly maximizing both the log likelihood and temporal smoothness. Temporal smoothness parameter $\beta$ ensures neighboring blocks are assigned to the same speaker. Thus, $\beta$ is a regularization parameter (switching penalty) that controls the trade-off between two objectives. For large value of $\beta\to\infty$, all speaker features are grouped together in a single cluster.

Given the model parameters (i.e., inverse covariance matrices) $\Theta_{i}$’s for each of the $K$ speaker clusters, we solve the optimization problem in Eqn. 2 that assigns the $T$ i-vectors, $X_{1}$, $X_{2}$,….., $X_{T}$ to $K$ clusters in a way that maximizes the likelihood of data while minimizing the number of times cluster assignment changes across time. Given $K$ potential cluster assignments for each of the $T$ data points, this combinatorial optimization problem has $KT$ possible mappings of feature vectors to speaker clusters. We assign each i-vector to a unique cluster using the approach presented in the Algorithm 1. This method is equivalent to finding the Viterbi path with minimum cost for the feature sequence of length $T$.

3.2 M-step: Toeplitz Graphical Lasso (Algorithm 2)

Each speaker cluster is modeled as a Gaussian inverse covariance matrix, $\Theta_{i}\in\mathbb{R}^{nbXnb}$ where $nb$ is the feature dimension. Since the inverse covariance matrix show the conditional independence structure between the variables, $\Theta_{i}$ defines a Markov Random Field (MRF) that encodes the structural representation present in all feature vectors of the i-th speaker cluster. Sparse graphical representations prevent overfitting in addition to fetching interpretable results [28]. In the M-step of EM algorithm, our objective is to estimate the $K$ inverse covariance matrices, $\mathbf{\Theta}=\{\Theta_{1},\Theta_{2},\cdots,\Theta_{K}\}$ using the cluster assignment sets, $\mathbb{C}=\{C_{1},C_{2},\cdots,C_{K}\}$, where $C_{i}\subset\{1,2,...,T\}$ obtained from the previous E-step. We can estimate each $\Theta_{i}$ in parallel thus saving execution time. The negative log likelihood of feature vector, $X_{t}$ to be assigned for i-th cluster with model parameter $\Theta_{i}$ is denoted as $N(X_{t},\Theta_{i})$. We can write it in terms of each $\Theta_{i}$ as follows [22]:

[TABLE]

where $|C_{i}|$ is the number of feature vectors assigned to the i-th cluster, $S_{i}$ is the empirical covariance of these feature vectors, and $\gamma$ is a constant independent of $\Theta_{i}$. Now, the M-step of EM algorithm becomes

[TABLE]

where $\mathbb{T}$ is a set of blockwise Toeplitz matrix. Eqn. 4 represents a convex optimization problem known as Toeplitz graphical lasso [22]. It is a type of graphical lasso problem with additional constraint of block Toeplitz structure on the inverse covariance matrices. Here, $\mathbb{\lambda}$ is a regularization parameter matrix for handling the trade-off between two objectives: (i) maximizing the log likelihood, and (ii) ensuring $\Theta_{i}$ to be sparse. For invertible matrix $S_{i}$, likelihood objective lead $\Theta_{i}$ to be near $S^{-1}_{i}$. The inverse covariance matrix, $\Theta_{i}$ is constrained to be block Toeplitz and $\mathbf{\lambda}$ is a $nbXnb$ matrix so that it can regularize each sub-block of $\Theta_{i}$ differently. A separate Toeplitz graphical lasso is solved for each speaker cluster at every iteration of the EM algorithm. Since we require several iterations of the EM algorithm before speaker clustering converges, a fast and efficient approach based on alternating direction method of multipliers (ADMM) is employed for this purpose [22]. For more details on solving this Toeplitz graphical lasso, please refer to Section 4.2 in [22].

4 Experiments, Results & Discussions

In this study, we focus on speaker clustering and hence ground-truth segmentation information is adopted for all experiments to avoid errors from SAD and speaker segmentation steps (see Fig. 1). We conduct evaluations on: (i) CRSS-PLTL, and (ii) two meetings subset of AMI corpus, as detailed below.

4.1 CRSS-PLTL Eval Set

It contain audio recordings of 80 minute PLTL sessions that are held once per week. PLTL is a student-led STEM education paradigm popular in US universities for undergraduate courses. Each session has 6-8 students plus a peer-leader who direct the discussions for collaborative problem solving among students [29]. CRSS-PLTL corpus [6] is a collection of audio recordings from five groups studying an undergraduate chemistry course over 11 weeks. Each participant in PLTL sessions wear a LENA device that collects the naturalistic audio [30]. We choose the channel corresponding to peer-leader for single-stream diarization studies reported in this paper. Our evaluation data consists of 80 minute audio with 8 speakers. We remove the overlapped speech, laughter, cough etc. from audio signal before passing it through our diarization system. Majority of the conversational turns are short (1-1.5s) for this data. Human annotators labeled the ground-truth speaker identity with start and end time stamps.

4.2 AMI Meeting Eval Set

Augmented Multi-party Interaction (AMI) corpus contain audio and video data recorded from meetings. Our AMI evaluation set consists of headset audio of two meetings namely IS1000a (26 min.) and IS1003b (27 min.) from the popular 12-meetings subset of AMI corpus [25, 31]. Both meetings involve four speakers discussing the design of a new remote control device.

4.3 Evaluations & Baseline Systems

Fig. 1 illustrate our diarization pipeline. We use frame-level 20-MFCC features extracted from 40ms windows with 10ms skip-rate. A UBM with 512 components is trained for i-vector extraction [13]. Given the short conversational-turns in PLTL, we train an i-vector extractor of dimension 75. For all experiments in this paper, we perform mean subtraction on i-vectors where the mean is computed from entire PLTL session/AMI meeting (see Fig. 1). Mean subtraction is followed by principal component analysis (PCA) for de-correlation and dimension reduction. We do PCA based dimension reduction to compare the performance over 21 and 51 components.

We recently proposed Mixtures of von Mises-Fisher Distributions (movMF) based speaker clustering for naturalistic audio data [21]. In this study, we choose cosine K-means [32] and movMF speaker clustering [21] as our baseline approaches. Unlike conventional K-means, cosine K-means projects the estimated cluster centroids onto unit hypersphere at the end of each M-step in the EM algorithm. Cosine K-means is a special case of movMF speaker clustering as discussed in [21]. When all mixture weights are equal and all concentration parameters are equal, the movMF speaker clustering becomes equivalent to cosine K-means. Feature vectors undergo length-normalization [33] before passing through cosine K-means [32] and movMF [21] baselines as these methods expect data lying on the unit hypersphere. On the other hand, length-normalization is optional for proposed TIC speaker clustering. We input either the mean-subtracted i-vectors or PCA processed i-vectors to the proposed TIC speaker clustering.

4.4 Results & Discussions

Diarization error rate (DER) is used for comparing the performance of different clustering approaches in this paper [34]. Mathematically, it is defined as:

[TABLE]

where $\alpha_{total}$ is the total scored time from reference annotations, $\alpha_{fa}$ is scored time for which a non-speech region is incorrectly marked as containing speech, $\alpha_{miss}$ is scored time for which a speech region is incorrectly marked as not containing speech, and $\alpha_{err}$ is the scored time for which a wrong speaker identity is assigned within a speech region. In this study, we focus on speaker clustering and hence ground-truth segmentation information was adopted in all experiments. Thus, $\alpha_{fa}$ and $\alpha_{miss}$ are zero. Here, DER is only dependent on speaker confusion error, $\alpha_{err}$. Overlapped speech, laughter and other miscellaneous vocalizations such as cough is removed from the audio signal before feeding into our diarization pipeline (see Fig. 1).

The proposed speaker clustering approach requires $\mathcal{O}(KT)$ time for grouping $T$ i-vectors into $K$ clusters. Hence, it is a scalable approach. Fig. 2 shows the DER comparison between proposed and baseline methods with various post-processing for PLTL data. Clearly, DER is reduced to almost half for proposed approach as compared to cosine K-means. The proposed method learns a correlation network modeled as MRF and hence it expects correlated data as input. After PCA, feature vectors are de-correlated and estimating the underlying correlation model using MRF is not accurate in this case. Consequently, as expected PCA degrades the performance of proposed approach on naturalistic PLTL data. The movMF is state-of-the-art for speaker clustering on PLTL data [21]. The proposed approach lead to 23.18% relative reduction in DER as compared to movMF speaker clustering. Significant DER reduction validate the efficacy of proposed Toeplitz Inverse Covariance (TIC)-based speaker clustering for naturalistic audio such as CRSS-PLTL.

Fig. 3 illustrate the DER comparison between proposed approach and baselines for two meetings subset of AMI corpus. Clearly, the best DER for both meetings are obtained by proposed approach. For both meetings, proposed approach fetch less than 1% DER showing its efficacy for real-world meeting data. For AMI meetings IS1000a and IS1003b, relative DER reduction is 29.37% and 9.21%, respectively. CRSS-PLTL data has higher levels and more varied forms of distortions as compared to AMI meeting data. Also, there are 8 speakers in PLTL data as compared to 4 in AMI meetings. These differences cause the DER to be in different ball park for both corpora.

5 Summary & Conclusions

In this paper, we leveraged the Toeplitz Inverse Covariance (TIC) estimation in speaker clustering task for naturalistic audio such as CRSS-PLTL corpus. Such audio data are rich in noise, overlapped-speech and reverberation in addition to short conversational turns (1s). The proposed approach accurately models the inner structure in i-vector features belonging to same speaker. It leads to enhanced clustering that eventually improves diarization performance. Each speaker cluster is characterized by a Markov Random Field (MRF) that captures its correlation structure. For robust speaker clustering task, the proposed approach iterates with alternating two steps of the standard Expectation Maximization (EM) algorithm: (i) E-step that assigns each speaker feature to a unique cluster; and (ii) M-step that update the model parameters using speaker clusters obtained in the E-step. We performed experiments on (i) CRSS-PLTL; and (iii) AMI meeting data. Cosine K-means and movMF were chosen as baseline clustering methods. Proposed TIC based speaker clustering fetch 43.22% relative reduction in DER for CRSS-PLTL data. AMI evaluation set has 29.37% (IS1000a) and 9.21% (IS1003b) relative reduction in DER over baseline. Significant improvements in DER show the efficacy of proposed speaker clustering for naturalistic audio streams.