Augmenting text for spoken language understanding with Large Language Models

TL;DR

This paper explores methods to improve spoken semantic parsing by augmenting training data with unpaired text, generated either from existing corpora or via Large Language Models, leading to significant performance gains.

Contribution

It introduces a novel approach of using LLMs to generate unpaired text for data augmentation in spoken language understanding, enhancing model performance across domains.

Findings

Unpaired text from existing corpora improves performance by 2% in existing domains.

Generated unpaired text from LLMs improves performance by 2.6% in new domains.

Using JAT and TTS with generated text enhances spoken semantic parsing accuracy.

Abstract

Spoken semantic parsing (SSP) involves generating machine-comprehensible parses from input speech. Training robust models for existing application domains represented in training data or extending to new domains requires corresponding triplets of speech-transcript-semantic parse data, which is expensive to obtain. In this paper, we address this challenge by examining methods that can use transcript-semantic parse data (unpaired text) without corresponding speech. First, when unpaired text is drawn from existing textual corpora, Joint Audio Text (JAT) and Text-to-Speech (TTS) are compared as ways to generate speech representations for unpaired text. Experiments on the STOP dataset show that unpaired text from existing and new domains improves performance by 2% and 30% in absolute Exact Match (EM) respectively. Second, we consider the setting when unpaired text is not available in existing textual corpora. We propose to prompt Large Language Models (LLMs) to generate unpaired text for existing and new domains. Experiments show that examples and words that co-occur with intents can be used to generate unpaired text with Llama 2.0. Using the generated text with JAT and TTS for spoken semantic parsing improves EM on STOP by 1.4% and 2.6% absolute for existing and new domains respectively.