Online Reasoning Video Segmentation with Just-in-Time Digital Twins

TL;DR

This paper introduces a novel online video reasoning segmentation framework using just-in-time digital twins, enabling efficient, multi-step reasoning without fine-tuning large language models, and presents a new benchmark for evaluation.

Contribution

The paper proposes a new agent framework that separates perception and reasoning, utilizing just-in-time digital twins to improve online video reasoning segmentation without LLM fine-tuning.

Findings

Effective handling of complex multi-step queries.

Scalable online video reasoning without fine-tuning.

Benchmark with 200 videos and 895 queries across multiple reasoning types.

Abstract

Reasoning segmentation (RS) aims to identify and segment objects of interest based on implicit text queries. As such, RS is a catalyst for embodied AI agents, enabling them to interpret high-level commands without requiring explicit step-by-step guidance. However, current RS approaches rely heavily on the visual perception capabilities of multimodal large language models (LLMs), leading to several major limitations. First, they struggle with queries that require multiple steps of reasoning or those that involve complex spatial/temporal relationships. Second, they necessitate LLM fine-tuning, which may require frequent updates to maintain compatibility with contemporary LLMs and may increase risks of catastrophic forgetting during fine-tuning. Finally, being primarily designed for static images or offline video processing, they scale poorly to online video data. To address these limitations, we propose an agent framework that disentangles perception and reasoning for online video RS without LLM fine-tuning. Our innovation is the introduction of a just-in-time digital twin concept, where -- given an implicit query -- a LLM plans the construction of a low-level scene representation from high-level video using specialist vision models. We refer to this approach to creating a digital twin as "just-in-time" because the LLM planner will anticipate the need for specific information and only request this limited subset instead of always evaluating every specialist model. The LLM then performs reasoning on this digital twin representation to identify target objects. To evaluate our approach, we introduce a new comprehensive video reasoning segmentation benchmark comprising 200 videos with 895 implicit text queries. The benchmark spans three reasoning categories (semantic, spatial, and temporal) with three different reasoning chain complexity.