Towards Agentic AI for Multimodal-Guided Video Object Segmentation

TL;DR

This paper introduces Multi-Modal Agent, an adaptive system leveraging large language models and specialized tools to improve multimodal-guided video object segmentation without task-specific training.

Contribution

The paper presents a novel agentic framework that dynamically adapts workflows for multimodal video segmentation using reasoning capabilities of LLMs.

Findings

Outperforms prior methods on RVOS and Ref-AVS tasks

Demonstrates flexibility and adaptability in multimodal segmentation

Reduces reliance on specialized training data

Abstract

Referring-based Video Object Segmentation is a multimodal problem that requires producing fine-grained segmentation results guided by external cues. Traditional approaches to this task typically involve training specialized models, which come with high computational complexity and manual annotation effort. Recent advances in vision-language foundation models open a promising direction toward training-free approaches. Several studies have explored leveraging these general-purpose models for fine-grained segmentation, achieving performance comparable to that of fully supervised, task-specific models. However, existing methods rely on fixed pipelines that lack the flexibility needed to adapt to the dynamic nature of the task. To address this limitation, we propose Multi-Modal Agent, a novel agentic system designed to solve this task in a more flexible and adaptive manner. Specifically, our method leverages the reasoning capabilities of large language models (LLMs) to generate dynamic workflows tailored to each input. This adaptive procedure iteratively interacts with a set of specialized tools designed for low-level tasks across different modalities to identify the target object described by the multimodal cues. Our agentic approach demonstrates clear improvements over prior methods on two multimodal-conditioned VOS tasks: RVOS and Ref-AVS.