Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos

TL;DR

Sa2VA is a unified multi-modal model that integrates SAM2 and LLaVA to enable dense, grounded understanding of images and videos, supporting diverse tasks with minimal instruction tuning.

Contribution

It introduces Sa2VA, the first model to unify static and dynamic visual understanding using a shared LLM token space, combining foundation video segmentation with vision-language modeling.

Findings

Achieves strong performance in referring video object segmentation

Supports a wide range of image and video tasks with minimal tuning

Provides a new dataset, Ref-SAV, with over 72k object expressions

Abstract

This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with MLLM, the advanced vision-language model, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2k video objects in the Ref-SAV datasets to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves strong performance across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications. In addition, Sa2VA can be easily extended into various VLMs, including Qwen-VL and Intern-VL, which can be updated with rapid process in current open-sourced VLMs. Code and models have been provided to the community.