Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition

TL;DR

This paper introduces a novel overlapped encoding separation method combined with serialized output training for multi-speaker ASR, enhancing separation and recognition accuracy by leveraging CTC and attention hybrid loss.

Contribution

It proposes the EncSep and GEncSep methods that improve multi-speaker speech recognition by better utilizing CTC and attention mechanisms in serialized output training.

Findings

Effective separation of single-speaker encoding from overlapped speech

CTC loss improves encoder representations in complex scenarios

GEncSep further enhances recognition performance

Abstract

Serialized output training (SOT) attracts increasing attention due to its convenience and flexibility for multi-speaker automatic speech recognition (ASR). However, it is not easy to train with attention loss only. In this paper, we propose the overlapped encoding separation (EncSep) to fully utilize the benefits of the connectionist temporal classification (CTC) and attention hybrid loss. This additional separator is inserted after the encoder to extract the multi-speaker information with CTC losses. Furthermore, we propose the serialized speech information guidance SOT (GEncSep) to further utilize the separated encodings. The separated streams are concatenated to provide single-speaker information to guide attention during decoding. The experimental results on LibriMix show that the single-speaker encoding can be separated from the overlapped encoding. The CTC loss helps to improve the encoder representation under complex scenarios. GEncSep further improved performance.