Learning Trajectory-Aware Multimodal Large Language Models for Video Reasoning Segmentation

TL;DR

TrajSeg introduces a bidirectional alignment approach in multimodal large language models to improve video object segmentation based on human instructions, enhancing trajectory perception and segmentation accuracy.

Contribution

It proposes a unified framework with bidirectional text-trajectory alignment and a frame-level content integration module for improved video reasoning segmentation.

Findings

Outperforms all existing methods on multiple datasets

Effective in perceiving object trajectories in dynamic videos

End-to-end trainable and simplified architecture

Abstract

The prosperity of Multimodal Large Language Models (MLLMs) has stimulated the demand for video reasoning segmentation, which aims to segment video objects based on human instructions. Previous studies rely on unidirectional and implicit text-trajectory alignment, which struggles with trajectory perception when faced with severe video dynamics. In this work, we propose TrajSeg, a simple and unified framework built upon MLLMs. Concretely, we introduce bidirectional text-trajectory alignment, where MLLMs accept grounding-intended (text-to-trajectory) and captioning-intended (trajectory-to-text) instructions. This way, MLLMs can benefit from enhanced correspondence and better perceive object trajectories in videos. The mask generation from trajectories is achieved via a frame-level content integration (FCI) module and a unified mask decoder. The former adapts the MLLM-parsed trajectory-level token to frame-specific information. The latter unifies segmentation for all frames into a single structure, enabling the proposed framework to be simplified and end-to-end trainable. Extensive experiments on referring and reasoning video segmentation datasets demonstrate the effectiveness of TrajSeg, which outperforms all video reasoning segmentation methods on all metrics. The code will be publicly available at https://github.com/haodi19/TrajSeg.