Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding

TL;DR

DEViL introduces a detector-empowered video large language model that enhances spatio-temporal grounding efficiency by offloading dense spatial tasks to a trained detector, enabling faster and more coherent video understanding.

Contribution

The paper proposes DEViL, a novel approach that combines a detector with a large language model to improve efficiency and coherence in spatio-temporal video grounding tasks.

Findings

DEViL achieves 43.1% m_vIoU on HC-STVG benchmark.

DEViL runs at 14.33 FPS, significantly faster than previous methods.

The approach maintains strong reasoning capabilities of the underlying LLM.

Abstract

Multimodal large language models (MLLMs) are rapidly expanding from general video understanding to finer-grained understanding such as spatio-temporal video grounding (STVG) and reasoning. In these tasks, an MLLM must localize the user-queried target in time and space and take the results as evidence for reasoning. Existing MLLM methods mainly follow two paradigms: (1) Direct Localization, which outputs STVG results with extra alignment modules or specialized decoders; and (2) Candidate-based Selection, which first constructs tube-level candidates and then selects the relevant one by an MLLM. However, both suffer from a serious efficiency bottleneck: the former incurs linearly growing decoding cost as the queried temporal span increases, while the latter relies on costly candidate construction. To break this bottleneck, we propose DEViL, a detector-empowered Video-LLM with a simple key idea: offloading dense spatial grounding from the MLLM to a fully parallelizable, well-trained detector. Specifically, DEViL distills the query into a detector-compatible reference-semantic token, which replaces the detector's text embedding to enable spatial grounding in a single pass. Then, we design temporal consistency regularization to match objects across frames and enforce their coherence over time. In this way, DEViL avoids long coordinate decoding and heavy candidate pipelines. Extensive experiments show that DEViL achieves strong performance (43.1% m_vIoU on HC-STVG) with superior efficiency (14.33 FPS), while preserving the general reasoning capacity of the MLLM backbone.