Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generation

TL;DR

This paper introduces a novel framework for creating datasets that combine contextual reasoning and paralinguistic understanding in speech-LLMs, enabling better empathetic and contextual speech processing.

Contribution

It presents the first framework for dataset generation that integrates in-the-wild speech data with paralinguistic labels and QA, enhancing speech-LLMs' reasoning capabilities.

Findings

Strong correlation between model evaluations and human datasets

Speech-LLMs show limitations in empathetic reasoning tasks

Framework enables training more robust speech-LLMs

Abstract

Current speech-LLMs exhibit limited capability in contextual reasoning alongside paralinguistic understanding, primarily due to the lack of Question-Answer (QA) datasets that cover both aspects. We propose a novel framework for dataset generation from in-the-wild speech data, that integrates contextual reasoning with paralinguistic information. It consists of a pseudo paralinguistic label-based data condensation of in-the-wild speech and LLM-based Contextual Paralinguistic QA (CPQA) generation. The effectiveness is validated by a strong correlation in evaluations of the Qwen2-Audio-7B-Instruct model on a dataset created by our framework and human-generated CPQA dataset. The results also reveal the speech-LLM's limitations in handling empathetic reasoning tasks, highlighting the need for such datasets and more robust models. The proposed framework is first of its kind and has potential in training more robust speech-LLMs with paralinguistic reasoning capabilities.