Speech Activity Detection Based on Multilingual Speech Recognition System

TL;DR

This paper introduces a multilingual speech recognition-based system for speech activity detection that leverages sequence discriminative training and decision fusion to improve robustness across languages and noise conditions.

Contribution

It proposes a novel SAD approach using multilingual ASR with LF-MMI training and decision fusion, outperforming existing baselines without requiring in-domain data.

Findings

Significantly better DetER on out-of-domain datasets.

Outperforms WebRTC, Phn Rec, and Pyannote baselines on Ester2 and LiveATC datasets.

Robustness to noise and language variability demonstrated.

Abstract

To better model the contextual information and increase the generalization ability of Speech Activity Detection (SAD) system, this paper leverages a multi-lingual Automatic Speech Recognition (ASR) system to perform SAD. Sequence discriminative training of Acoustic Model (AM) using Lattice-Free Maximum Mutual Information (LF-MMI) loss function, effectively extracts the contextual information of the input acoustic frame. Multi-lingual AM training, causes the robustness to noise and language variabilities. The index of maximum output posterior is considered as a frame-level speech/non-speech decision function. Majority voting and logistic regression are applied to fuse the language-dependent decisions. The multi-lingual ASR is trained on 18 languages of BABEL datasets and the built SAD is evaluated on 3 different languages. On out-of-domain datasets, the proposed SAD model shows significantly better performance with respect to baseline models. On the Ester2 dataset, without using any in-domain data, this model outperforms the WebRTC, phoneme recognizer based VAD (Phn Rec), and Pyannote baselines (respectively by 7.1, 1.7, and 2.7% absolute) in Detection Error Rate (DetER) metrics. Similarly, on the LiveATC dataset, this model outperforms the WebRTC, Phn Rec, and Pyannote baselines (respectively by 6.4, 10.0, and 3.7% absolutely) in DetER metrics.