Real-time Speech Interruption Analysis: From Cloud to Client Deployment

TL;DR

This paper presents a real-time speech interruption detection model that is optimized for deployment on client devices, improving accuracy, reducing size and complexity, and estimating environmental benefits.

Contribution

It introduces a novel, efficient speech interruption model with enhanced performance and significantly reduced size and complexity for client deployment, along with environmental impact analysis.

Findings

Improved TPR at 1% FPR from 50.9% to 68.3%.

Reduced model size from 222.7 MB to 9.3 MB.

Lowered complexity from 31.2 GMACS to 4.3 GMACS.

Abstract

Meetings are an essential form of communication for all types of organizations, and remote collaboration systems have been much more widely used since the COVID-19 pandemic. One major issue with remote meetings is that it is challenging for remote participants to interrupt and speak. We have recently developed the first speech interruption analysis model, which detects failed speech interruptions, shows very promising performance, and is being deployed in the cloud. To deliver this feature in a more cost-efficient and environment-friendly way, we reduced the model complexity and size to ship the WavLM_SI model in client devices. In this paper, we first describe how we successfully improved the True Positive Rate (TPR) at a 1% False Positive Rate (FPR) from 50.9% to 68.3% for the failed speech interruption detection model by training on a larger dataset and fine-tuning. We then shrank the model size from 222.7 MB to 9.3 MB with an acceptable loss in accuracy and reduced the complexity from 31.2 GMACS (Giga Multiply-Accumulate Operations per Second) to 4.3 GMACS. We also estimated the environmental impact of the complexity reduction, which can be used as a general guideline for large Transformer-based models, and thus make those models more accessible with less computation overhead.