Audiopedia: Audio QA with Knowledge

TL;DR

Audiopedia introduces a new audio question answering task that combines audio comprehension with external knowledge reasoning, proposing a framework to enhance large audio language models' performance on knowledge-intensive audio tasks.

Contribution

The paper defines a novel knowledge-intensive Audio Question Answering task and proposes a generic framework to improve large audio language models' reasoning capabilities.

Findings

Benchmarking shows current models perform suboptimally on Audiopedia tasks.

The proposed framework with AEL and KA2LM components improves model performance.

First work to address advanced audio understanding with knowledge reasoning.

Abstract

In this paper, we introduce Audiopedia, a novel task called Audio Question Answering with Knowledge, which requires both audio comprehension and external knowledge reasoning. Unlike traditional Audio Question Answering (AQA) benchmarks that focus on simple queries answerable from audio alone, Audiopedia targets knowledge-intensive questions. We define three sub-tasks: (i) Single Audio Question Answering (s-AQA), where questions are answered based on a single audio sample, (ii) Multi-Audio Question Answering (m-AQA), which requires reasoning over multiple audio samples, and (iii) Retrieval-Augmented Audio Question Answering (r-AQA), which involves retrieving relevant audio to answer the question. We benchmark large audio language models (LALMs) on these sub-tasks and observe suboptimal performance. To address this, we propose a generic framework that can be adapted to any LALM, equipping them with knowledge reasoning capabilities. Our framework has two components: (i) Audio Entity Linking (AEL) and (ii) Knowledge-Augmented Audio Large Multimodal Model (KA2LM), which together improve performance on knowledge-intensive AQA tasks. To our knowledge, this is the first work to address advanced audio understanding via knowledge-intensive tasks like Audiopedia.