A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning

TL;DR

This paper proposes an audio-visual wake word spotting system that leverages visual lip information and neural network pruning to improve performance and reduce complexity in noisy environments, suitable for TV applications.

Contribution

It introduces an iterative fine-tuning neural network pruning method based on the lottery ticket hypothesis for compact multi-modal wake word systems.

Findings

Audio-visual system outperforms single-modality systems in noisy conditions.

LTH-IF pruning significantly reduces model size and computation without performance loss.

The approach is effective for real-world TV wake-up scenarios.

Abstract

Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission. In this paper, we investigate on designing a compact audio-visual WWS system by utilizing visual information to alleviate the degradation. Specifically, in order to use visual information, we first encode the detected lips to fixed-size vectors with MobileNet and concatenate them with acoustic features followed by the fusion network for WWS. However, the audio-visual model based on neural networks requires a large footprint and a high computational complexity. To meet the application requirements, we introduce a neural network pruning strategy via the lottery ticket hypothesis in an iterative fine-tuning manner (LTH-IF), to the single-modal and multi-modal models, respectively. Tested on our in-house corpus for audio-visual WWS in a home TV scene, the proposed audio-visual system achieves significant performance improvements over the single-modality (audio-only or video-only) system under different noisy conditions. Moreover, LTH-IF pruning can largely reduce the network parameters and computations with no degradation of WWS performance, leading to a potential product solution for the TV wake-up scenario.