Object-centric Video Question Answering with Visual Grounding and Referring

TL;DR

This paper introduces a new VideoLLM model with object referring and grounding capabilities, a novel spatial-temporal overlay module, and a curated dataset, achieving state-of-the-art results across multiple video understanding benchmarks.

Contribution

The paper presents a novel VideoLLM with object-centric reasoning, a spatial-temporal overlay module, and a new dataset, advancing multimodal video understanding and interaction.

Findings

Outperforms baselines on 12 benchmarks across 6 tasks

Demonstrates robustness in object-centric video reasoning

Enables multimodal, multiround video interactions

Abstract

Video Large Language Models (VideoLLMs) have recently demonstrated remarkable progress in general video understanding. However, existing models primarily focus on high-level comprehension and are limited to text-only responses, restricting the flexibility for object-centric, multiround interactions. In this paper, we make three contributions: (i) we address these limitations by introducing a VideoLLM model, capable of performing both object referring for input and grounding for output in video reasoning tasks, i.e., allowing users to interact with videos using both textual and visual prompts; (ii) we propose STOM (Spatial-Temporal Overlay Module), a novel approach that propagates arbitrary visual prompts input at any single timestamp to the remaining frames within a video; (iii) we present VideoInfer, a manually curated object-centric video instruction dataset featuring questionanswering pairs that require reasoning. We conduct comprehensive experiments on VideoInfer and other existing benchmarks across video question answering and referring object segmentation. The results on 12 benchmarks of 6 tasks show that our proposed model consistently outperforms baselines in both video question answering and segmentation, underscoring its robustness in multimodal, object-centric video and image understanding. Project page: https://qirui-chen.github.io/RGA3-release/.