SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models

TL;DR

This paper introduces SAMA, a comprehensive framework including a large dataset, a novel model, and a benchmark for multi-turn referential grounded video chat, significantly advancing fine-grained video understanding and grounded dialogue capabilities.

Contribution

The paper presents SAMA-239K dataset, SAMA model with spatio-temporal aggregation and grounding, and SAMA-Bench benchmark, enabling unified multi-turn video referential understanding.

Findings

SAMA achieves state-of-the-art results on grounding benchmarks.

SAMA performs strongly on multi-turn video chat tasks.

The dataset and benchmark facilitate comprehensive evaluation of video LMMs.

Abstract

Achieving fine-grained spatio-temporal understanding in videos remains a major challenge for current Video Large Multimodal Models (Video LMMs). Addressing this challenge requires mastering two core capabilities: video referring understanding, which captures the semantics of video regions, and video grounding, which segments object regions based on natural language descriptions. However, most existing approaches tackle these tasks in isolation, limiting progress toward unified, referentially grounded video interaction. We identify a key bottleneck in the lack of high-quality, unified video instruction data and a comprehensive benchmark for evaluating referentially grounded video chat. To address these challenges, we contribute in three core aspects: dataset, model, and benchmark. First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically curated to enable joint learning of video referring understanding, grounding, and multi-turn video chat. Second, we propose the SAMA model, which incorporates a versatile spatio-temporal context aggregator and a Segment Anything Model to jointly enhance fine-grained video comprehension and precise grounding capabilities. Finally, we establish SAMA-Bench, a meticulously designed benchmark consisting of 5,067 questions from 522 videos, to comprehensively evaluate the integrated capabilities of Video LMMs in multi-turn, spatio-temporal referring understanding and grounded dialogue. Extensive experiments and benchmarking results show that SAMA not only achieves strong performance on SAMA-Bench but also sets a new state-of-the-art on general grounding benchmarks, while maintaining highly competitive performance on standard visual understanding benchmarks.