Neural Network based End-to-End Query by Example Spoken Term Detection

TL;DR

This paper introduces a neural network end-to-end framework for query by example spoken term detection, outperforming traditional DTW-based methods by jointly optimizing feature extraction and pattern matching.

Contribution

It presents the first fully neural network-based end-to-end system for QbE-STD, replacing separate feature extraction and matching stages with joint optimization.

Findings

Multilingual bottleneck features improve with more training languages.

CNN-based matching outperforms DTW-based matching with bottleneck features.

End-to-end training significantly improves detection performance.

Abstract

This paper focuses on the problem of query by example spoken term detection (QbE-STD) in zero-resource scenario. State-of-the-art approaches primarily rely on dynamic time warping (DTW) based template matching techniques using phone posterior or bottleneck features extracted from a deep neural network (DNN). We use both monolingual and multilingual bottleneck features, and show that multilingual features perform increasingly better with more training languages. Previously, it has been shown that the DTW based matching can be replaced with a CNN based matching while using posterior features. Here, we show that the CNN based matching outperforms DTW based matching using bottleneck features as well. In this case, the feature extraction and pattern matching stages of our QbE-STD system are optimized independently of each other. We propose to integrate these two stages in a fully neural network based end-to-end learning framework to enable joint optimization of those two stages simultaneously. The proposed approaches are evaluated on two challenging multilingual datasets: Spoken Web Search 2013 and Query by Example Search on Speech Task 2014, demonstrating in each case significant improvements.