CUSIDE-array: A Streaming Multi-Channel End-to-End Speech Recognition System with Realistic Evaluations

TL;DR

This paper introduces CUSIDE-array, a streaming multi-channel end-to-end speech recognition system that enhances out-of-distribution generalization and realistic evaluation by integrating chunking, future context simulation, and decoding into neural beamforming.

Contribution

It presents a novel CUSIDE-array method combining recent techniques with neural beamforming for streaming ME2E ASR, addressing OOD generalization and realistic evaluation challenges.

Findings

Achieves superior streaming results in ID and OOD tests

Enables real-time processing with 402ms latency

Improves OOD generalization through back-end pre-training and fine-tuning

Abstract

Recently multi-channel end-to-end (ME2E) ASR systems have emerged. While streaming single-channel end-to-end ASR has been extensively studied, streaming ME2E ASR is limited in exploration. Additionally, recent studies call attention to the gap between in-distribution (ID) and out-of-distribution (OOD) tests and doing realistic evaluations. This paper focuses on two research problems: realizing streaming ME2E ASR and improving OOD generalization. We propose the CUSIDE-array method, which integrates the recent CUSIDE methodology (Chunking, Simulating Future Context and Decoding) into the neural beamformer approach of ME2E ASR. It enables streaming processing of both front-end and back-end with a total latency of 402ms. The CUSIDE-array ME2E models are shown to achieve superior streaming results in both ID and OOD tests. Realistic evaluations confirm the advantage of CUSIDE-array in its capability to consume single-channel data to improve OOD generalization via back-end pre-training and ME2E fine-tuning.