ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation

TL;DR

ViCaS introduces a comprehensive dataset combining detailed video captions with pixel-level object segmentation, enabling unified evaluation of high-level understanding and precise localization in videos.

Contribution

The paper presents ViCaS, a novel dataset that unifies high-level video captioning and pixel-precise segmentation with grounded language annotations, along with a new benchmark and model architecture.

Findings

ViCaS dataset contains thousands of videos with detailed captions and segmentation masks.

Proposed model architecture effectively handles both holistic understanding and pixel-level segmentation.

Evaluation measures demonstrate the dataset's utility for advancing multimodal video understanding.

Abstract

Recent advances in multimodal large language models (MLLMs) have expanded research in video understanding, primarily focusing on high-level tasks such as video captioning and question-answering. Meanwhile, a smaller body of work addresses dense, pixel-precise segmentation tasks, which typically involve category-guided or referral-based object segmentation. Although both directions are essential for developing models with human-level video comprehension, they have largely evolved separately, with distinct benchmarks and architectures. This paper aims to unify these efforts by introducing ViCaS, a new dataset containing thousands of challenging videos, each annotated with detailed, human-written captions and temporally consistent, pixel-accurate masks for multiple objects with phrase grounding. Our benchmark evaluates models on both holistic/high-level understanding and language-guided, pixel-precise segmentation. We also present carefully validated evaluation measures and propose an effective model architecture that can tackle our benchmark. Project page: https://ali2500.github.io/vicas-project/