Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder

TL;DR

This paper introduces a unified multi-speaker encoder that jointly learns representations for diarization, separation, and ASR, improving performance on overlapping speech tasks by leveraging shared representations and multi-level semantic information.

Contribution

The novel UME architecture unifies multiple speech tasks using a shared encoder and residual weighted-sum encoding, capturing task interdependencies for improved accuracy.

Findings

Significant reduction in diarization error rates on LibriMix datasets.

Outperforms previous methods in speaker diarization accuracy.

Enhances multi-task learning for overlapping speech recognition.

Abstract

This paper presents a unified multi-speaker encoder (UME), a novel architecture that jointly learns representations for speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) tasks using a shared speech foundational encoder. We leverage the hidden representations from multiple layers of UME as a residual weighted-sum encoding (RWSE) to effectively use information from different semantic levels, contributing to bottom-up alignment between tasks. This joint training approach captures the inherent interdependencies among the tasks, enhancing overall performance on overlapping speech data. Our evaluations demonstrate that UME substantially improves over the single-task baselines dedicated to SD, SS, and multi-speaker ASR on LibriMix evaluation sets. Notably, for SD, UME outperforms the previous studies, achieving diarization error rates of 1.37% and 2.29% on Libri2Mix and Libri3Mix evaluation sets, respectively.

Equations

$$X=\sum _{c=1}^C{Y}^c\odot S^c+N,$$

$$H_{(l)}={\mathrm{SpeechEnc}}_{(l)}(X).$$

$$H^{\text{ws}}=\sum _{l=1}^L{\omega }_{(l)}^{\text{task}}{H}_{(l)},$$

$$H^{\text{enc}}=H^{\text{ws}}+H_{(l=L)}.$$

$${\mathcal{L}}_{\text{diar}}=\underset{\pi \in \mathcal{P}}{\min }\sum _{c=1}^C\mathrm{BCE}(Y^{\pi (c)},P^c(H^{\text{enc}})),$$

$$H^{\text{sep}}=\mathrm{ConvEnc}(X).$$

$$H^{\text{concat}}=\mathrm{concat}(H^{\text{sep}},\mathrm{upsample}(H^{\text{enc}})).$$

$$E=\mathrm{TCN}(\mathrm{Con}{\mathrm{v}}_{1\times 1}(\mathrm{LayerNorm}(H^{\text{concat}}))).$$

$$M^c=\sigma ([\mathrm{Con}{\mathrm{v}}_{1\times 1}(\mathrm{PReLU}(E)){]}_c),$$

$$D^c=H^{\text{concat}}\odot M^c.$$

$${\hat{S}}^c=\mathrm{Decoder}(D^c),$$

$${\mathcal{L}}_{\text{sep}}=\underset{\pi \in \mathcal{P}}{\min }\left(-10\sum _{c=1}^C{\log }_{10}\left(\frac{{\Vert \frac{⟨{\hat{S}}^c,S^{\pi (c)}⟩S^{\pi (c)}}{{\Vert S^{\pi (c)}\Vert }^2}\Vert }^2}{{\Vert {\hat{S}}^c-\frac{⟨{\hat{S}}^c,S^{\pi (c)}⟩S^{\pi (c)}}{{\Vert S^{\pi (c)}\Vert }^2}\Vert }^2}\right)\right),$$

$$H_c^{\text{asr}}={\mathrm{SpeakerEnc}}_{\text{SD}}^c(H^{\text{enc}}).$$

$${\mathcal{L}}_{\text{asr}}=\underset{\pi \in \mathcal{P}}{\min }\sum _{c=1}^C[$$

$$+(1-{\lambda }_{\text{ctc}}){\mathcal{L}}_{\text{att}}(W^{\pi (c)},H_c^{\text{asr}})],$$

$${\mathcal{L}}_{\text{all}}={\lambda }_{\text{diar}}{\mathcal{L}}_{\text{diar}}+{\lambda }_{\text{sep}}{\mathcal{L}}_{\text{sep}}+{\lambda }_{\text{asr}}{\mathcal{L}}_{\text{asr}}.$$

Full text

††publicationid: pubid:

© 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

I Introduction

Speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR) are tasks of great importance that aim to comprehend and answer the question “who spoke what and when,” with applications to transcribing meetings and interviews, among others. Previous studies in SD [1, 2, 3], SS [4, 5, 6], and multi-speaker ASR [7, 8, 9, 10] have focused primarily on improving the quality of single-task models that operate independently on acoustic information to separate or label speaker segments and transcribe the text in a speech-processing system [11, 12, 13]. A key limitation of training tasks independently is that inter-dependencies cannot be leveraged.

Most existing frameworks [13, 14, 15] address this limitation by unifying speech-processing architectures [16, 17]. These architectures consist of either a joint ASR/SD [18, 19, 20], SS/ASR [21, 22, 23], or a SD/SS [24, 25] task following a fixed optimal order that can vary depending on the target scene scenario [26, 27, 28, 29]. These different target scenes suggest that we solve these tasks jointly, independent of the order, so all these tasks can benefit from each other. Lately, there has been a shift towards employing pre-trained speech foundation models (SFMs) [30, 31, 32, 33, 34] in end-to-end (E2E) systems, which effectively learn useful representations for various speech processing tasks [35]. However they do not work well on multi-speaker conversation recognition. Additionally, it has been demonstrated that different layers encode different types of information in SD and ASR tasks [35, 32]. Preliminary observation from these studies shows that intermediate layers of the encoder extract a rich hierarchy of information, e.g., in WavLM large [32], initial layers and last layers are more critical for SD and ASR tasks. Therefore, it makes sense to utilize multiple layers to jointly optimize all SD, SS, and ASR tasks effectively. The question, therefore, naturally arises: can we build a unified model that leverages all encoder layers to optimize performance across multiple tasks?

Motivated by the potential of SFMs and E2E speech processing, we propose a unified multi-speaker encoder (UME), a novel E2E speech-processing framework. The proposed framework is generalizable to use any SFM, E2E SD, SS and multi-speaker ASR task and jointly optimizes all these tasks into a single network with multitask learning to minimize the error accumulation for a speech processing framework. Additionally, by extracting features from all the layers of the OWSM (open Whisper style speech model [36]) v3.1 encoder and using it as a residual weighted-sum encoding (RWSE), we can learn better hidden representations from the encoder layers. We hypothesize that RWSE introduces information exchange and better bottom-up alignment to all the tasks from different semantic levels. We argue that UME framework should provide a shared representation space for SD, SS and multi-speaker ASR tasks and preferably have strong generalizability and learnability. We conduct extensive experiments on different design choices of UME on Libri2Mix & Libri3Mix [37] datasets. Our key contributions are:

• •

We propose a unified speech-processing framework to jointly optimize the performance of SD, SS, and multi-speaker ASR tasks with hidden representations of the SFM encoder.

• •

We propose using RWSE of the pre-trained SFM encoder layers to unify and optimize the performance across diverse speech processing tasks.

• •

We demonstrate the effectiveness of our framework on two-speaker and three-speaker overlapped speech and obtain substantial performance improvement in each diarization, separation, and multi-speaker ASR task.

II Unified Multi-Speaker Encoder (UME)

Figure 1 shows the overall framework of UME. It leverages the hidden representations through an RWSE of intermediate layers, which act as a bridge between SD, SS, and multi-speaker ASR tasks. This enables a comprehensive and detailed interaction from each layer of the SFM encoder. Note that our goal is not to develop a new encoder or speech processing tasks; in principle, one can apply any SFM encoder, SD, SS, or multi-speaker ASR tasks.

We start with the $T$-length single-channel input speech mixture. $X=\{x_{t}\in\mathbb{R}|t=1,\cdots,T\}$ of $C$ speakers. We define the input speech mixture in an anechoic condition by:

[TABLE]

where, $S^{c}=\{s^{c}_{t}\in\mathbb{R}|t=1,...,T\}$ is the clean speech signal of speaker $c$, $Y^{c}=\{y^{c}_{t}\in[0,1]\mid t=1,\cdots,T\}$ is a binary speech activity sequence indicating whether speaker $c$ is talking at time $t$ or not, and $N=\{n_{t}\in\mathbb{R}\mid t=1,\cdots,T\}$ is the additive noise signal. Together, the activity sequences $\{Y^{c}\}_{c=1}^{C}$ form the ground truth speaker label for the speaker diarization (SD) task described in Section II-B.

II-A Speech Foundation Model Encoder

We selected OWSMv3.1 [36] as the shared encoder due to its widespread recognition, reproducibility, open-source availability, and fast, efficient encoding capabilities. We can note that OWSM was trained on single-speaker speech-to-text tasks (i.e., no speaker tasks in pre-training). However, we can still adapt it to our multi-speaker setup. The speech encoder is a stack of $L$ E-Branchformer [38] encoder layers that transforms the input speech mixture $X$ of $C$ speakers into a $D^{\text{enc}}$-dimensional subsampled $T^{\text{enc}}(<T)$-length hidden state representations $H_{(l)}=\{\mathbf{h}_{(t,l)}\in\mathbb{R}^{D^{\text{enc}}}|t=1,\cdots,T^{\text{enc}}\}$, where $l$ is a layer index from 1 to $L$. The simplified speech encoder is given by:

[TABLE]

To integrate into task-specific models, all encoder layers are combined into a single feature vector via a weighted sum [35]:

[TABLE]

where $\omega^{\text{task}}_{(l)}$ are softmax-normalized learnable weights that scale the representations from different encoder layers depending on each task. Finally, RWSE is introduced by adding $H^{\text{ws}}$ to the last encoder layer ($l=L$) to amplify the influence of the final layer according to [32] across tasks during training.

[TABLE]

II-B Speaker Diarization Task

Given the robust performance of E2E Neural Diarization (EEND) [2] with permutation invariant training (PIT) in estimating multi-speaker activities within an E2E framework, we adopt EEND for the SD task in the proposed UME E2E framework. SD involves predicting speaker activity as binary multi-class labels by estimating the speaker label sequence. We encode the hidden state representations $H^{\text{enc}}$ from the speech encoder and map the speaker activity probabilities $P^{c}(H^{\text{enc}})\in\{0,1\}^{C}$ for speaker $c$ using a linear layer. We train EEND with PIT using speaker activity probabilities and the target speaker activity labels. We optimize the binary cross entropy-based (BCE) diarization loss ($\mathcal{L}_{\text{diar}}$) below:

[TABLE]

where $\pi(c)$ is the mapping of $c$-th element under permutation $\pi$ and $\mathcal{P}$ is the set of all permutations over $c=\{1,\cdots,C\}$ and $Y^{\pi(c)}$ is the permuted reference of speaker labels.

II-C Speech Separation Task

We incorporate Conv-TasNet [5] into our framework for its demonstrated adaptability in E2E-SS [39] task. While having suboptimal performance, its simple architecture makes it a well-known time-domain speech separation model capable of predicting speech signals using a fully convolutional encoder, separator, and decoder network. The input mixture is encoded via a 1-D convolutional encoder which is given by:

[TABLE]

We concatenate upsampled RWSE representations ($H^{\text{enc}}$) from Section II-A with $H^{\text{sep}}$ and obtain $D^{\text{sep}}$-dimensional subsampled $T^{\text{sep}}(<T)$-length hidden state representations $H^{\text{concat}}=\{\mathbf{h}_{t}^{\text{concat}}\in\mathbb{R}^{D^{\text{sep}}}\mid t=1,\cdots,T^{\text{sep}}\}$ which is given by:

[TABLE]

These representations are then processed using stacked 1-D dilated temporal convolutional networks (TCNs) to extract the embedding sequence $E=\{\mathbf{e}_{t}\in\mathbb{R}^{K}\mid t=1,\cdots,T^{\text{sep}}\}$ in (8):

[TABLE]

The separation network then estimates a mask sequence $M^{c}=\{\mathbf{m}^{c}_{t}\in[0,1]^{O}|t=1,\cdots,T^{\text{sep}}\}$ in (9) for each source:

[TABLE]

where it computes the source-specific representation sequence $D^{c}=\{\mathbf{d}^{c}_{t}\in\mathbb{R}^{J}\mid t=1,\dots,T^{\text{sep}}\}$ by applying the masks using element-wise multiplication $\odot$ in (10).

[TABLE]

Finally, a 1-D transposed convolutional decoder reconstructs the time-domain waveform for each source $\hat{S}^{c}=\{\hat{s}_{t}^{c}\in\mathbb{R}\mid t=1,\dots,T\}$ in (11) and optimizes the $\mathrm{SI-SDR}$ loss ($\mathcal{L}_{\text{sep}}$) in (12).

[TABLE]

[TABLE]

II-D Multi-speaker ASR Task

The multi-speaker ASR task, as adopted from [9], extends a joint connectionist temporal classification (CTC)/attention-based framework to recognize speech from multiple speakers. We encode the input hidden state representations (see Section II-A) from the speech encoder. Subsequently, each speaker’s speech is extracted through $C$ speaker-differentiating encoder blocks (SpeakerEnc${}_{\text{SD}}^{c}$). These speaker-dependent features are then transformed into $D^{\text{asr}}$-dimensional subsampled $T^{\text{asr}}(<T^{\text{enc}})$-length hidden state representations $H^{\text{asr}}_{c}=\{\mathbf{h}^{\text{asr}}_{(t,c)}\in\mathbb{R}^{D^{\text{asr}}}\mid t=1,\cdots,T^{\text{asr}}\}$, where $c=\{1,\cdots,C\}$ enumerates speakers.

[TABLE]

The attention-based decoder generates a speaker-specific $U$-length output token sequence $W^{c}=\{w^{c}_{u}\in\mathcal{V}\mid u=1,\cdots,U\}$, where $w^{c}_{u}$ is an output token at position $u$ in the vocabulary $\mathcal{V}$ for speakers $c=1,\cdots,C$. Specifically, PIT (see Section II-B) is employed to control the reference sequences $W^{c}$ permutation $\pi$ using the CTC loss ($\mathcal{L}_{\text{ctc}}$) immediately after the encoder. Finally, the loss for the multi-speaker ASR ($\mathcal{L}_{\text{asr}}$) task is optimized using CTC and cross-entropy loss of the attention decoder ($\mathcal{L}_{\text{att}}$):

[TABLE]

We optimize the final loss function as a weighted sum of $\mathcal{L}_{\text{diar}}$ in (5), $\mathcal{L}_{\text{sep}}$ in (12) and $\mathcal{L}_{\text{asr}}$ in (14). $\lambda_{\text{diar}}$, $\lambda_{\text{sep}}$, and $\lambda_{\text{asr}}$ are the weighting hyperparameters which are optimized empirically.

[TABLE]

Thus, we can integrate all diarization, separation, and ASR tasks in a unified multi-speaker-encoder architecture.

III Experiments

III-A Dataset and evaluation metrics

In UME, we aim to optimize all three tasks: diarization, separation, and multi-speaker ASR, using multi-task learning in a unified framework. We require three ground truths to objectively evaluate performance, i.e., diarization labels, separated sources, and text for each speaker. While real-world multiparty datasets [40, 41] exist for diarization-only tasks, they often need separated sources and the number of speakers to be known for ASR. Therefore, we employ simulated conversation-like open-source datasets for training and evaluation of all three tasks. For training, we use LibriMix, which combines LibriSpeech samples with and without WHAM! noise, supporting two-speaker (Libri2Mix) and three-speaker (Libri3Mix) mixtures. We adopt a 16kHz sampling rate, the “mixboth” and “mixclean” method with 100% overlap. Moreover, we selected the “max” mode in LibriMix, as the ASR task is unfeasible in the “min” mode due to the truncation of speech signals on minimum-length sequences. We evaluate our framework using the Libri2Mix and Libri3Mix datasets [37] with a complete 100% overlap, as well as the LibriSpeech2Mix and LibriSpeech3Mix datasets [10], which include a partial random overlap of at least 0.5 seconds.

We evaluate diarization using the diarization error rate (DER%) [42] with a 0.0s, 0.25s collar tolerance and 11-frame median filtering. For separation, we report three metrics: STOI (dB) [43], SDR (dB) [44] and SI-SNR (dB) [5]. Multi-speaker ASR performance is measured using WER with the optimum permutation following the prior studies in [9] and [10]. Unlike [13], our WER computation is independent of the diarization branch i.e., diarization frame information is not used during the multi-speaker ASR decoding process.

III-B Implementation details

UME employs the pre-trained supervised SFM encoder OWSMv3.1 [36] (medium) as a shared feature extractor for all tasks. We use learnable weights with RWSE (see Section II-A) to optimize all OWSMv3.1 layers jointly. For diarization following the hyperparameters reported in [2] (see Section II-B), we input the RWSE features (frame length: 400, frameshift: 640 samples) into a 1-layer RNN attractor (hidden size: 1024). For separation (see Section II-C), we concatenate 1024-dimensional OWSMv3.1 hidden representations with 256-dimensional Conv-TasNet [5] encoded features. Since OWSMv3.1 has a 40 ms frameshift, we up-sample its features for time alignment, as in (7) before feeding them into three TCN blocks with eight convolutional layers (hidden size: 512). For multi-speaker ASR (see Section II-D), we project OWSMv3.1 features to 128 dimensions, then process them with a post-encoder having four SpeakerEnc${}_{\text{SD}}^{c}$ transformer blocks (2048 linear units, 256 input dimension) and a convolutional layer with a subsampling factor of four. It is jointly optimized ($\lambda_{\text{ctc}}$=0.2 & $\lambda_{\text{att}}$=0.8) with six transformer-based decoder blocks (2048 linear units, 256 input dimension) and a CTC layer following [9]. During training in UME, we initialize the encoder parameters with the pre-trained OWSMv3.1 medium encoder and fine-tune the encoder layers for 70 epochs, while all task-specific parameters have a flat start (i.e., no parameter initialization for task-specific layers) and are trained for 70 epochs. In the ASR-initialized UME version, the ASR model is pre-trained separately for 30 epochs before integration. We use the AdamW optimizer with an initial learning rate of $4\cdot 10^{-4}$ (empirically tuned) and weight decay of $1\cdot 10^{-6}$ warmed up for 10,000 steps. Training on four A100 80GB GPUs took six days, with an average batch size of 44, dynamically adjusted using the ESPnet [48] numel batch type. For task-specific weighting, we use an equal-weight scalarization approach [49], assigning $\lambda_{\text{asr}}$=0.33, $\lambda_{\text{diar}}$=0.33, $\lambda_{\text{sep}}$=0.34 to balance tasks. This approach assumes that the tasks are cooperative rather than conflicting. While we also explored a two-stage weight optimization strategy inspired by [50], however, it led to performance degradation.

IV Main Results

Tables I, II, and III show the performance of UME compared with previous works on downstream single task frameworks. Moreover, we also compare our results and report the findings by explicitly setting the multi-task learning weights of the individual tasks to zero in our unified framework for an unbiased comparison, providing more insights about the flexibility of our proposed method. In the following sections, we discuss the experimental results in detail.

IV-A Multi-speaker ASR results

For the multi-speaker ASR task, we first input the OWSMv3.1 extracted features through a shallow speaker-differentiating encoder trained with CTC, attention, and PIT losses without using the SD and SS tasks ($\lambda_{\text{asr}}$=1.0). Similar to a previous study [9] which is our reproducible baseline, we initialized the SpeakerEnc${}_{\text{SD}}^{c}$ blocks (see Section II-D) with a pre-trained model from the ESPnet recipe for training stability. For the multi-speaker ASR task in UME, we observe that the initialization of the ASR model provides training stability and outperforms the strong baselines in Table I both for 100% overlap (Libri2Mix & Libri3Mix) and partial overlap task (LibriSpeech2Mix & LibriSpeech3Mix). On the “mixclean” evaluation sets (Libri2Mix and Libri3Mix), UME with RWSE achieves a WER of 6.4% and 15.9%, respectively. Furthermore, our evaluation of the “mixboth” evaluation set validates the effectiveness of the proposed method in noisy conditions. Our experiments also indicate that initializing the three-speaker model with a pre-trained two-speaker model is essential, as training without such initialization consistently resulted in divergence. Notably, the UME framework was trained using the “mixboth” Libri2Mix and Libri3Mix training set, which combines two-speaker and three-speaker mixtures with WHAM! noise in 100% overlap setting but also evaluated on the LibriSpeech2Mix and LibriSpeech3Mix evaluation set containing only clean speech with partial overlap. This demonstrates its superior generalization ability across datasets with varying data modeling characteristics.

IV-B End-to-end speaker diarization results

Table II presents the results for the UME in the SD task, which outperforms WavLM [32], achieving a DER of 1.37% in a 100% overlapped task setting for Libri2Mix. Furthermore, UME also achieved state-of-the-art results on Libri3Mix. Notably, WavLM is trained using overlapped speech mixtures, whereas OWSMv3.1 [36] is trained solely on clean speech. Despite this, OWSMv3.1, adapted as the multi-speaker encoder having an improved architecture, outperforms WavLM. We believe that the additional training losses from SS and multi-speaker ASR tasks provide additional granularity during the training in the multi-task learning framework.

IV-C End-to-end speech separation results

Unlike previous studies [35, 32] which report the separation results in “min mode”, UME requires overlapped mixtures in “max mode” during the training process due to the unification of the ASR task, as discussed in Section III-A. For this reason, we evaluate the UME on 100% overlapped mixtures in “max mode” with our fully reproduced Conv-TasNet model following the similar setup in Section III-B without the concatenated features. Experimental results in Table III demonstrate consistent improvement compared to the separation-only tasks, indicating that concatenating encoded features with the upsampled hidden representations of the OWSMv3.1 encoder in the TCN block (see Section II-C) improves separation performance (see Figures 2 & 3), resulting in an improved performance in recovering the speaker activity without using an additional diarization branch.

V Conclusion

In this paper, we propose UME, a unified framework for end-to-end speech processing, which integrates speaker diarization, speech separation, and multi-speaker ASR with a residual weighted-sum encoding (RWSE) of the intermediate encoder layers. UME substantially outperforms strong baselines and previous works and achieves state-of-the-art performance on the speaker diarization task. In the future, we plan to apply this framework to more challenging multi-speaker scenarios like CHiME-6 [13]. We are also interested in extending it to a multilingual UME.