End-to-End Spoken Language Understanding Without Full Transcripts

TL;DR

This paper presents end-to-end spoken language understanding models that directly map speech to semantic entities, reducing the need for full transcripts and demonstrating strong performance even with limited annotation.

Contribution

It introduces speech-to-entities models trained solely on semantic annotations, showing they can effectively recognize entities without full transcriptions.

Findings

Both models effectively skip non-entity words with minimal performance loss.

Attention model handles re-ordering of entities with only 2% F1 score degradation.

Models trained on entities alone perform comparably to those trained on full transcripts.

Abstract

An essential component of spoken language understanding (SLU) is slot filling: representing the meaning of a spoken utterance using semantic entity labels. In this paper, we develop end-to-end (E2E) spoken language understanding systems that directly convert speech input to semantic entities and investigate if these E2E SLU models can be trained solely on semantic entity annotations without word-for-word transcripts. Training such models is very useful as they can drastically reduce the cost of data collection. We created two types of such speech-to-entities models, a CTC model and an attention-based encoder-decoder model, by adapting models trained originally for speech recognition. Given that our experiments involve speech input, these systems need to recognize both the entity label and words representing the entity value correctly. For our speech-to-entities experiments on the ATIS corpus, both the CTC and attention models showed impressive ability to skip non-entity words: there was little degradation when trained on just entities versus full transcripts. We also explored the scenario where the entities are in an order not necessarily related to spoken order in the utterance. With its ability to do re-ordering, the attention model did remarkably well, achieving only about 2% degradation in speech-to-bag-of-entities F1 score.