Transformer-based encoder-encoder architecture for Spoken Term Detection

TL;DR

This paper introduces a Transformer-based encoder-encoder architecture for spoken term detection that leverages shared embeddings and outperforms LSTM-based baselines on English and Czech datasets.

Contribution

The novel encoder-encoder architecture with convolutional, upsampling, and attention masking modifications improves spoken term detection performance.

Findings

Outperforms LSTM-based baseline methods

Effective on English and Czech datasets

Utilizes shared embedding space for scoring

Abstract

The paper presents a method for spoken term detection based on the Transformer architecture. We propose the encoder-encoder architecture employing two BERT-like encoders with additional modifications, including convolutional and upsampling layers, attention masking, and shared parameters. The encoders project a recognized hypothesis and a searched term into a shared embedding space, where the score of the putative hit is computed using the calibrated dot product. In the experiments, we used the Wav2Vec 2.0 speech recognizer, and the proposed system outperformed a baseline method based on deep LSTMs on the English and Czech STD datasets based on USC Shoah Foundation Visual History Archive (MALACH).