Advancing VAD Systems Based on Multi-Task Learning with Improved Model Structures

TL;DR

This paper introduces a multi-task learning approach with improved model structures for voice activity detection, enhancing noise robustness and accuracy in both real-time and offline speech recognition systems.

Contribution

It proposes semantic VAD models based on RWKV and SAN-M architectures, improving noise robustness and detection accuracy over traditional DFSMN-based systems.

Findings

Real-time semantic VAD with RWKV reduces CER by 7.0% and DCF by 26.1%.

Offline VAD with SAN-M reduces CER by 4.4% and DCF by 18.6%.

Both models improve NRR by over 3% compared to baseline systems.

Abstract

In a speech recognition system, voice activity detection (VAD) is a crucial frontend module. Addressing the issues of poor noise robustness in traditional binary VAD systems based on DFSMN, the paper further proposes semantic VAD based on multi-task learning with improved models for real-time and offline systems, to meet specific application requirements. Evaluations on internal datasets show that, compared to the real-time VAD system based on DFSMN, the real-time semantic VAD system based on RWKV achieves relative decreases in CER of 7.0\%, DCF of 26.1\% and relative improvement in NRR of 19.2\%. Similarly, when compared to the offline VAD system based on DFSMN, the offline VAD system based on SAN-M demonstrates relative decreases in CER of 4.4\%, DCF of 18.6\% and relative improvement in NRR of 3.5\%.