Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning

TL;DR

This paper presents a novel approach to adapt large language models for speech recognition, especially disordered speech, using reinforcement learning with custom rewards, offering an alternative to traditional fine-tuning.

Contribution

It introduces a method combining audio token integration and reinforcement learning to improve LLM adaptation to disordered speech, surpassing supervised fine-tuning in certain settings.

Findings

Reinforcement learning with custom rewards improves adaptation to disordered speech.

The method outperforms supervised fine-tuning in specific speech recognition scenarios.

The approach offers a new tuning strategy for large language models in speech tasks.

Abstract

We introduce a large language model (LLM) capable of processing speech inputs and show that tuning it further with reinforcement learning on human preference (RLHF) enables it to adapt better to disordered speech than traditional fine-tuning. Our method replaces low-frequency text tokens in an LLM's vocabulary with audio tokens and enables the model to recognize speech by fine-tuning it on speech with transcripts. We then use RL with rewards based on syntactic and semantic accuracy measures generalizing the LLM further to recognize disordered speech. While the resulting LLM does not outperform existing systems for speech recognition, we find that tuning with reinforcement learning using custom rewards leads to substantially better performance than supervised fine-tuning of the language model, specifically when adapting to speech in a different setting. This presents a compelling alternative tuning strategy for speech recognition using large language models.