OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios

TL;DR

OmniGround is a new comprehensive benchmark for spatio-temporal video grounding that addresses current limitations by providing diverse, complex real-world data and a systematic evaluation framework, enabling better model robustness.

Contribution

The paper introduces OmniGround, a large-scale, diverse benchmark with a novel annotation pipeline and evaluation framework, advancing the assessment of models on real-world complex scenarios.

Findings

Models experience a 10.4% performance drop on complex scenes.

PG-TAF framework improves grounding accuracy by over 25%.

OmniGround enables more robust and realistic evaluation of STVG models.

Abstract

Spatio-Temporal Video Grounding (STVG) aims to localize target objects in videos based on natural language descriptions. Despite recent advances in Multimodal Large Language Models, a significant gap remains between current models and real-world demands involving diverse objects and complex queries. We attribute this to limited benchmark scope, causing models to exhibit category bias, oversimplified reasoning, and poor linguistic robustness. To address these limitations, we introduce OmniGround, a comprehensive benchmark with 3,475 videos spanning 81 categories and complex real-world queries. We propose the Forward-Backward-Refinement annotation pipeline that combines multi-directional tracking with intelligent error correction for high-quality labels. We further introduce DeepSTG, a systematic evaluation framework quantifying dataset quality across four complementary dimensions beyond superficial statistics. Evaluations reveal performance average drop of 10.4% on complex real-world scenes, particularly with small/occluded objects and intricate spatial relations. Motivated by these, we propose PG-TAF, a training-free two-stage framework decomposing STVG into high-level temporal grounding and fine-grained spatio-temporal propagation. Experiments demonstrate PG-TAF achieves 25.6% and 35.6% improvements in m\_tIoU and m\_vIoU on OmniGround with consistent gains across four benchmarks.