Introducing Semantics into Speech Encoders

TL;DR

This paper presents an unsupervised method to incorporate semantic information from large language models into self-supervised speech encoders, significantly enhancing their spoken language understanding capabilities without requiring labeled audio data.

Contribution

It introduces a task-agnostic, unsupervised approach to embed semantic information into speech encoders, reducing reliance on costly labeled transcriptions.

Findings

Over 10% improvement in intent classification accuracy

Modest gains in named entity resolution and slot filling

Over 2% increase in spoken question answering FF1 score

Abstract

Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM. These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain. We propose a task-agnostic unsupervised way of incorporating semantic information from LLMs into self-supervised speech encoders without labeled audio transcriptions. By introducing semantics, we improve existing speech encoder spoken language understanding performance by over 10\% on intent classification, with modest gains in named entity resolution and slot filling, and spoken question answering FF1 score by over 2\%. Our unsupervised approach achieves similar performance as supervised methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.