Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

TL;DR

This paper introduces TGS-Agent, a reasoning-based approach for referring audio-visual segmentation that mimics human reasoning, leveraging explicit object understanding and multimodal analysis to improve segmentation accuracy without pixel-level supervision.

Contribution

The paper proposes a novel explicit reasoning framework with Ref-Thinker and a new benchmark R2-AVSBench for better evaluation of reasoning-intensive referring AVS tasks.

Findings

Achieves state-of-the-art results on Ref-AVSBench

Introduces a new benchmark R2-AVSBench with diverse references

Demonstrates effectiveness of explicit reasoning over latent embedding methods

Abstract

Referring Audio-Visual Segmentation (Ref-AVS) aims to segment target objects in audible videos based on given reference expressions. Prior works typically rely on learning latent embeddings via multimodal fusion to prompt a tunable SAM/SAM2 decoder for segmentation, which requires strong pixel-level supervision and lacks interpretability. From a novel perspective of explicit reference understanding, we propose TGS-Agent, which decomposes the task into a Think-Ground-Segment process, mimicking the human reasoning procedure by first identifying the referred object through multimodal analysis, followed by coarse-grained grounding and precise segmentation. To this end, we first propose Ref-Thinker, a multimodal language model capable of reasoning over textual, visual, and auditory cues. We construct an instruction-tuning dataset with explicit object-aware think-answer chains for Ref-Thinker fine-tuning. The object description inferred by Ref-Thinker is used as an explicit prompt for Grounding-DINO and SAM2, which perform grounding and segmentation without relying on pixel-level supervision. Additionally, we introduce R\textsuperscript{2}-AVSBench, a new benchmark with linguistically diverse and reasoning-intensive references for better evaluating model generalization. Our approach achieves state-of-the-art results on both standard Ref-AVSBench and proposed R\textsuperscript{2}-AVSBench. Code will be available at https://github.com/jasongief/TGS-Agent.