E.M.Ground: A Temporal Grounding Vid-LLM with Holistic Event Perception and Matching

TL;DR

E.M.Ground is a novel Video Large Language Model designed for more accurate temporal video grounding by capturing holistic event semantics, reducing noise, and improving event matching through innovative token and feature aggregation techniques.

Contribution

It introduces a <event> token, smoothing techniques, and multi-grained feature aggregation to enhance event perception and matching in temporal video grounding tasks.

Findings

Outperforms state-of-the-art Vid-LLMs on benchmark datasets

Achieves significant improvements in event localization accuracy

Demonstrates robustness to noise and information loss

Abstract

Despite recent advances in Video Large Language Models (Vid-LLMs), Temporal Video Grounding (TVG), which aims to precisely localize time segments corresponding to query events, remains a significant challenge. Existing methods often match start and end frames by comparing frame features with two separate tokens, relying heavily on exact timestamps. However, this approach fails to capture the event's semantic continuity and integrity, leading to ambiguities. To address this, we propose E.M.Ground, a novel Vid-LLM for TVG that focuses on holistic and coherent event perception. E.M.Ground introduces three key innovations: (i) a special <event> token that aggregates information from all frames of a query event, preserving semantic continuity for accurate event matching; (ii) Savitzky-Golay smoothing to reduce noise in token-to-frame similarities across timestamps, improving prediction accuracy; (iii) multi-grained frame feature aggregation to enhance matching reliability and temporal understanding, compensating for compression-induced information loss. Extensive experiments on benchmark datasets show that E.M.Ground consistently outperforms state-of-the-art Vid-LLMs by significant margins.