Listening Between the Frames: Bridging Temporal Gaps in Large Audio-Language Models

TL;DR

This paper introduces TimeAudio, a novel method that enhances large audio-language models with precise temporal perception and efficient long audio understanding, enabling better performance on fine-grained temporal tasks.

Contribution

We propose TimeAudio, which incorporates temporal markers, absolute time-aware encoding, and a segment-level token merging module to improve temporal reasoning and long audio comprehension in LALMs.

Findings

Strong performance on dense captioning, temporal grounding, and timeline speech summarization.

Effective temporal localization and reasoning capabilities demonstrated.

New dataset and metrics for evaluating fine-grained temporal tasks.

Abstract

Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data. To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance. Evaluations show strong performance across a variety of fine-grained tasks, such as dense captioning, temporal grounding, and timeline speech summarization, demonstrating TimeAudio's robust temporal localization and reasoning capabilities.