Acoustic span embeddings for multilingual query-by-example search

TL;DR

This paper introduces acoustic span embeddings (ASE) for multilingual query-by-example speech search, enabling faster and more accurate search across multiple languages and arbitrary query lengths, especially in low-resource settings.

Contribution

It generalizes acoustic word embeddings to spans of words and demonstrates their effectiveness for multilingual QbE with arbitrary-length queries.

Findings

ASE-based search is significantly faster than DTW-based search.

ASE outperforms previous state-of-the-art results on QUESST 2015.

Multilingual ASE effectively handles low-resource and unseen languages.

Abstract

Query-by-example (QbE) speech search is the task of matching spoken queries to utterances within a search collection. In low- or zero-resource settings, QbE search is often addressed with approaches based on dynamic time warping (DTW). Recent work has found that methods based on acoustic word embeddings (AWEs) can improve both performance and search speed. However, prior work on AWE-based QbE has primarily focused on English data and with single-word queries. In this work, we generalize AWE training to spans of words, producing acoustic span embeddings (ASE), and explore the application of ASE to QbE with arbitrary-length queries in multiple unseen languages. We consider the commonly used setting where we have access to labeled data in other languages (in our case, several low-resource languages) distinct from the unseen test languages. We evaluate our approach on the QUESST 2015 QbE tasks, finding that multilingual ASE-based search is much faster than DTW-based search and outperforms the best previously published results on this task.