Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech

TL;DR

This paper demonstrates that large language models can understand paralinguistic speech cues without fine-tuning, by using an end-to-end system that aligns speech and text prompts to produce empathetic responses.

Contribution

It introduces a novel framework that enables frozen LLMs to perceive paralinguistic speech aspects through a trained speech encoder, without modifying the LLM's weights.

Findings

System produces more empathetic responses to expressive speech.

Outperforms several baseline methods in response quality.

First to induce frozen LLMs to understand non-linguistic speech cues.

Abstract

This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights. We utilize an end-to-end system with a speech encoder, which is trained to produce token embeddings such that the LLM's response to an expressive speech prompt is aligned with its response to a semantically matching text prompt that has also been conditioned on the user's speaking style. This framework enables the encoder to generate tokens that capture both linguistic and paralinguistic information and effectively convey them to the LLM, even when the LLM's weights remain completely frozen. To the best of our knowledge, our work is the first to explore how to induce a frozen LLM to understand more than just linguistic content from speech inputs in a general interaction setting. Experiments demonstrate that our system is able to produce higher quality and more empathetic responses to expressive speech prompts compared to several baselines.