VISA: Reasoning Video Object Segmentation via Large Language Models

TL;DR

This paper introduces ReasonVOS, a new task for video object segmentation that involves complex reasoning with world knowledge, and presents VISA, a model leveraging large language models for this purpose, supported by a large benchmark dataset.

Contribution

The paper proposes ReasonVOS, a novel segmentation task requiring reasoning, and introduces VISA, a multi-modal LLM-based model with a new benchmark dataset for training and evaluation.

Findings

VISA effectively handles complex reasoning in video segmentation.

VISA outperforms existing methods on multiple datasets.

The benchmark facilitates future research in reasoning-based segmentation.

Abstract

Existing Video Object Segmentation (VOS) relies on explicit user instructions, such as categories, masks, or short phrases, restricting their ability to perform complex video segmentation requiring reasoning with world knowledge. In this paper, we introduce a new task, Reasoning Video Object Segmentation (ReasonVOS). This task aims to generate a sequence of segmentation masks in response to implicit text queries that require complex reasoning abilities based on world knowledge and video contexts, which is crucial for structured environment understanding and object-centric interactions, pivotal in the development of embodied AI. To tackle ReasonVOS, we introduce VISA (Video-based large language Instructed Segmentation Assistant), to leverage the world knowledge reasoning capabilities of multi-modal LLMs while possessing the ability to segment and track objects in videos with a mask decoder. Moreover, we establish a comprehensive benchmark consisting of 35,074 instruction-mask sequence pairs from 1,042 diverse videos, which incorporates complex world knowledge reasoning into segmentation tasks for instruction-tuning and evaluation purposes of ReasonVOS models. Experiments conducted on 8 datasets demonstrate the effectiveness of VISA in tackling complex reasoning segmentation and vanilla referring segmentation in both video and image domains. The code and dataset are available at https://github.com/cilinyan/VISA.