Instruction-Following Speech Recognition

TL;DR

This paper introduces an instruction-following speech recognition model trained from scratch that can understand and execute diverse free-form instructions, enabling flexible speech tasks without relying on large language models.

Contribution

It presents a novel training approach for speech recognition models to follow natural language instructions, expanding capabilities beyond traditional transcription.

Findings

Model trained on Librispeech can interpret and execute instructions without pre-trained modules.

Enables tasks like transcript manipulation and summarization.

Provides selective transcription for privacy and safety.

Abstract

Conventional end-to-end Automatic Speech Recognition (ASR) models primarily focus on exact transcription tasks, lacking flexibility for nuanced user interactions. With the advent of Large Language Models (LLMs) in speech processing, more organic, text-prompt-based interactions have become possible. However, the mechanisms behind these models' speech understanding and "reasoning" capabilities remain underexplored. To study this question from the data perspective, we introduce instruction-following speech recognition, training a Listen-Attend-Spell model to understand and execute a diverse set of free-form text instructions. This enables a multitude of speech recognition tasks -- ranging from transcript manipulation to summarization -- without relying on predefined command sets. Remarkably, our model, trained from scratch on Librispeech, interprets and executes simple instructions without requiring LLMs or pre-trained speech modules. It also offers selective transcription options based on instructions like "transcribe first half and then turn off listening," providing an additional layer of privacy and safety compared to existing LLMs. Our findings highlight the significant potential of instruction-following training to advance speech foundation models.