Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models

TL;DR

This paper presents a resource-efficient multimodal system that detects device-directed speech without trigger phrases by combining audio and speech recognition signals using large language models, requiring minimal training data.

Contribution

It introduces a novel multimodal approach utilizing low-rank adaptation and prefix tuning for resource-efficient device-directed speech detection with large language models.

Findings

Multimodal approach achieves lower EERs than unimodal baselines.

System requires only a small amount of training data (80k examples or less).

Low-dimensional audio representations outperform high-dimensional ones.

Abstract

Interactions with virtual assistants typically start with a trigger phrase followed by a command. In this work, we explore the possibility of making these interactions more natural by eliminating the need for a trigger phrase. Our goal is to determine whether a user addressed the virtual assistant based on signals obtained from the streaming audio recorded by the device microphone. We address this task by combining 1-best hypotheses and decoder signals from an automatic speech recognition system with acoustic representations from an audio encoder as input features to a large language model (LLM). In particular, we are interested in data and resource efficient systems that require only a small amount of training data and can operate in scenarios with only a single frozen LLM available on a device. For this reason, our model is trained on 80k or less examples of multimodal data using a combination of low-rank adaptation and prefix tuning. We compare the proposed system to unimodal baselines and show that the multimodal approach achieves lower equal-error-rates (EERs), while using only a fraction of the training data. We also show that low-dimensional specialized audio representations lead to lower EERs than high-dimensional general audio representations.