Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt

TL;DR

This paper introduces TimePro-RL, a framework that enhances large audio-language models' ability to perceive fine-grained temporal details using audio-side time prompts and reinforcement learning.

Contribution

It proposes a novel audio-side time prompt method combined with RL to improve temporal perception in large audio-language models, addressing a key limitation.

Findings

TimePro-RL significantly improves performance in audio grounding, sound event detection, and dense captioning.

Encoding timestamps as embeddings and interleaving them enhances temporal understanding.

Reinforcement learning optimizes the model's temporal alignment capabilities.

Abstract

Large Audio-Language Models (LALMs) enable general audio understanding and demonstrate remarkable performance across various audio tasks. However, these models still face challenges in temporal perception (e.g., inferring event onset and offset), leading to limited utility in fine-grained scenarios. To address this issue, we propose Audio-Side Time Prompt and leverage Reinforcement Learning (RL) to develop the TimePro-RL framework for fine-grained temporal perception. Specifically, we encode timestamps as embeddings and interleave them within the audio feature sequence as temporal coordinates to prompt the model. Furthermore, we introduce RL following Supervised Fine-Tuning (SFT) to directly optimize temporal alignment performance. Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning, validating its robust effectiveness.