LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models

TL;DR

This paper introduces LibriSQA, a new dataset and framework for spoken question answering using large language models, demonstrating improved multimodal understanding and alignment between speech and text.

Contribution

The paper presents a novel LibriSQA dataset with 107k pairs and a lightweight end-to-end framework for SQA, advancing multimodal LLM capabilities.

Findings

Framework achieves significant results in SQA tasks.

Reformulating ASR as SQA enhances speech-text understanding.

Empirical evidence shows improved multimodal alignment.

Abstract

While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multimodal functionalities, especially for the Spoken Question Answering (SQA) task which necessitates precise alignment and deep interaction between speech and text features. To address the SQA challenge on LLMs, we initially curated the free-form and open-ended LibriSQA dataset from Librispeech, comprising Part I with natural conversational formats and Part II encompassing multiple-choice questions followed by answers and analytical segments. Both parts collectively include 107k SQA pairs that cover various topics. Given the evident paucity of existing speech-text LLMs, we propose a lightweight, end-to-end framework to execute the SQA task on the LibriSQA, witnessing significant results. By reforming ASR into the SQA format, we further substantiate our framework's capability in handling ASR tasks. Our empirical findings bolster the LLMs' aptitude for aligning and comprehending multimodal information, paving the way for the development of universal multimodal LLMs. The dataset and demo can be found at https://github.com/ZihanZhaoSJTU/LibriSQA.