DisTime: Distribution-based Time Representation for Video Large Language Models

TL;DR

DisTime introduces a novel distribution-based temporal representation framework for Video-LLMs, improving temporal localization and understanding by using continuous embeddings and a new dataset with extensive temporally grounded annotations.

Contribution

The paper presents DisTime, a lightweight framework with a distribution-based time decoder and encoder, along with a large annotated dataset, to enhance temporal comprehension in Video-LLMs.

Findings

Achieves state-of-the-art results on multiple temporal tasks.

Creates InternVid-TG, a dataset with 1.25 million temporally grounded events.

Maintains competitive performance in Video QA tasks.

Abstract

Despite advances in general video understanding, Video Large Language Models (Video-LLMs) face challenges in precise temporal localization due to discrete time representations and limited temporally aware datasets. Existing methods for temporal expression either conflate time with text-based numerical values, add a series of dedicated temporal tokens, or regress time using specialized temporal grounding heads. To address these issues, we introduce DisTime, a lightweight framework designed to enhance temporal comprehension in Video-LLMs. DisTime employs a learnable token to create a continuous temporal embedding space and incorporates a Distribution-based Time Decoder that generates temporal probability distributions, effectively mitigating boundary ambiguities and maintaining temporal continuity. Additionally, the Distribution-based Time Encoder re-encodes timestamps to provide time markers for Video-LLMs. To overcome temporal granularity limitations in existing datasets, we propose an automated annotation paradigm that combines the captioning capabilities of Video-LLMs with the localization expertise of dedicated temporal models. This leads to the creation of InternVid-TG, a substantial dataset with 1.25M temporally grounded events across 179k videos, surpassing ActivityNet-Caption by 55 times. Extensive experiments demonstrate that DisTime achieves state-of-the-art performance across benchmarks in three time-sensitive tasks while maintaining competitive performance in Video QA tasks. Code and data are released at https://github.com/josephzpng/DisTime.