Bidirectional Representations for Low Resource Spoken Language Understanding

TL;DR

This paper introduces a bidirectional speech representation model using masked language modeling, improving low-resource spoken language understanding and achieving state-of-the-art results with efficient class attention mechanisms.

Contribution

It proposes a novel bidirectional encoding approach for speech, leveraging masked language modeling and class attention for better low-resource spoken language understanding.

Findings

Better pre-fine-tuning representations than comparable models

State-of-the-art performance on Fluent Speech Command dataset

Effective in low-data regimes

Abstract

Most spoken language understanding systems use a pipeline approach composed of an automatic speech recognition interface and a natural language understanding module. This approach forces hard decisions when converting continuous inputs into discrete language symbols. Instead, we propose a representation model to encode speech in rich bidirectional encodings that can be used for downstream tasks such as intent prediction. The approach uses a masked language modelling objective to learn the representations, and thus benefits from both the left and right contexts. We show that the performance of the resulting encodings before fine-tuning is better than comparable models on multiple datasets, and that fine-tuning the top layers of the representation model improves the current state of the art on the Fluent Speech Command dataset, also in a low-data regime, when a limited amount of labelled data is used for training. Furthermore, we propose class attention as a spoken language understanding module, efficient both in terms of speed and number of parameters. Class attention can be used to visually explain the predictions of our model, which goes a long way in understanding how the model makes predictions. We perform experiments in English and in Dutch.