Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

TL;DR

This paper introduces a reinforcement learning approach that grounds speech reasoning in explicit audio timestamps, improving model focus, reasoning quality, and performance on speech-based tasks.

Contribution

It presents a novel timestamp grounding method for large audio-language models, enhancing reasoning fidelity and task performance in speech understanding.

Findings

Timestamp grounding improves model attention to relevant audio segments.

Grounding enhances reasoning behaviors like exploration and verification.

The approach outperforms zero-shot and non-grounded fine-tuning methods.

Abstract

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chains remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.