Towards Semi-Supervised Semantics Understanding from Speech

TL;DR

This paper introduces a semi-supervised framework for speech-based semantics understanding that leverages pretrained models and performs well under noisy conditions with limited labeled data.

Contribution

The paper presents a novel semi-supervised approach combining pretrained ASR and language models for speech semantics understanding, addressing noise robustness and limited data challenges.

Findings

Achieves parity with oracle text-based models in noisy environments

Demonstrates effectiveness with limited labeled data

Introduces the slots edit F1 score as a new evaluation metric

Abstract

Much recent work on Spoken Language Understanding (SLU) falls short in at least one of three ways: models were trained on oracle text input and neglected the Automatics Speech Recognition (ASR) outputs, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. We proposed a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed speech to address these. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU corpus. In parallel, we identified two inadequate settings under which SLU models have been tested: noise-robustness and E2E semantics evaluation. We tested the proposed framework under realistic environmental noises and with a new metric, the slots edit F1 score, on two public SLU corpora. Experiments show that our SLU framework with speech as input can perform on par with those with oracle text as input in semantics understanding, while environmental noises are present, and a limited amount of labeled semantics data is available.