MAR3: Multi-Agent Recognition, Reasoning, and Reflection for Reference Audio-Visual Segmentation

TL;DR

MAR3 introduces a novel, training-free multi-agent framework for reference audio-visual segmentation that explicitly recognizes expression difficulty, dominant modality, and incorporates reflective validation, leading to superior performance.

Contribution

The paper presents MAR3, a new multi-agent, training-free framework that improves reference audio-visual segmentation by explicitly modeling expression difficulty, modality dominance, and using reflective validation.

Findings

Achieves 69.2% J&F on Ref-AVSBench, surpassing SOTA by 3.4%.

Introduces a Consensus Multimodal Recognition mechanism for better modality understanding.

Develops a Reflective Learning Segmentation mechanism for iterative correction of segmentation masks.

Abstract

Reference Audio-Visual Segmentation (Ref-AVS) aims to segment objects in audible videos based on multimodal cues in reference expressions. Previous methods overlook the explicit recognition of expression difficulty and dominant modality in multimodal cues, over-rely on the quality of the instruction-tuning dataset for object reasoning, and lack reflective validation of segmentation results, leading to erroneous mask predictions. To address these issues, in this paper, we propose a novel training-free Multi-Agent Recognition, Reasoning, and Reflection framework to achieve high-quality Reference Audio-Visual Segmentation, termed MAR3. Incorporating the sociological Delphi theory to achieve robust analysis, a Consensus Multimodal Recognition mechanism is proposed that enables LLM agents to explicitly recognize the difficulty of reference expressions and the dominant modality of multimodal cues. Based on our modality-dominant difficulty rule, we propose an adaptive Collaborative Object Reasoning strategy to reliably reason about the referred object. To further ensure precise mask prediction, we develop a Reflective Learning Segmentation mechanism, in which a check agent examines intermediate segmentation results and iteratively corrects the object text prompt of the segment agent. Experiments demonstrate that MAR3 achieves superior performance (69.2% in J&F) on the Ref-AVSBench dataset, outperforming SOTA by 3.4% absolutely.