Advancing Complex Video Object Segmentation via Progressive Concept Construction

TL;DR

This paper introduces Segment Concept (SeC), a novel VOS framework leveraging high-level object-centric representations and large vision-language models to improve segmentation in complex, concept-rich video scenarios.

Contribution

SeC shifts from feature matching to progressive concept construction using LVLMs, and introduces SeCVOS, a benchmark for high-level reasoning in video object segmentation.

Findings

SeC outperforms state-of-the-art methods on SeCVOS and standard benchmarks.

Achieves 11.8-point improvement over SAM 2.1 on SeCVOS.

Demonstrates robustness in scenarios with appearance variations and scene changes.

Abstract

We propose Segment Concept (SeC), a concept-driven video object segmentation (VOS) framework that shifts from conventional feature matching to the progressive construction and utilization of high-level, object-centric representations. SeC employs Large Vision-Language Models (LVLMs) to integrate visual cues across diverse frames, constructing robust conceptual priors. To balance semantic reasoning with computational overhead, SeC forwards the LVLMs only when a new scene appears, injecting concept-level features at those points. To rigorously assess VOS methods in scenarios demanding high-level conceptual reasoning and robust semantic understanding, we introduce the Semantic Complex Scenarios Video Object Segmentation benchmark (SeCVOS). SeCVOS comprises 160 manually annotated multi-scenario videos designed to challenge models with substantial appearance variations and dynamic scene transformations. Empirical evaluations demonstrate that SeC substantially outperforms state-of-the-art approaches, including SAM 2 and its advanced variants, on both SeCVOS and standard VOS benchmarks. In particular, SeC achieves an 11.8-point improvement over SAM 2.1 on SeCVOS, establishing a new state-of-the-art in concept-aware VOS.