Streaming on-device detection of device directed speech from voice and touch-based invocation

TL;DR

This paper introduces a novel streaming on-device detection method for device-directed speech that handles both voice and touch invocations, improving accuracy and efficiency for virtual assistant activation.

Contribution

The paper presents the first streaming approach capable of detecting device-directed speech from multiple invocation types using a new TCN-based decision layer for on-device deployment.

Findings

Streaming TCN outperforms alternatives in accuracy and speed.

All models show minimal accuracy degradation compared to invocation-specific models.

The approach reduces runtime peak-memory by up to 33% compared to LSTM-based methods.

Abstract

When interacting with smart devices such as mobile phones or wearables, the user typically invokes a virtual assistant (VA) by saying a keyword or by pressing a button on the device. However, in many cases, the VA can accidentally be invoked by the keyword-like speech or accidental button press, which may have implications on user experience and privacy. To this end, we propose an acoustic false-trigger-mitigation (FTM) approach for on-device device-directed speech detection that simultaneously handles the voice-trigger and touch-based invocation. To facilitate the model deployment on-device, we introduce a new streaming decision layer, derived using the notion of temporal convolutional networks (TCN) [1], known for their computational efficiency. To the best of our knowledge, this is the first approach that can detect device-directed speech from more than one invocation type in a streaming fashion. We compare this approach with streaming alternatives based on vanilla Average layer, and canonical LSTMs, and show: (i) that all the models show only a small degradation in accuracy compared with the invocation-specific models, and (ii) that the newly introduced streaming TCN consistently performs better or comparable with the alternatives, while mitigating device undirected speech faster in time, and with (relative) reduction in runtime peak-memory over the LSTM-based approach of 33% vs. 7%, when compared to a non-streaming counterpart.