VideoOrion: Tokenizing Object Dynamics in Videos

TL;DR

VideoOrion introduces a novel approach to capturing and encoding object dynamics in videos as semantic tokens, enabling efficient and explicit object modeling for improved video understanding tasks.

Contribution

The paper proposes a new method for extracting and encoding object dynamics into semantic tokens, improving over prior downsampling and resampling techniques in Video-LLMs.

Findings

Achieves competitive results on video question answering benchmarks.

Enables explicit object modeling with minimal computational cost.

Provides a more natural and disentangled semantic representation of videos.

Abstract

We present VideoOrion, a Video Large Language Model (Video-LLM) that explicitly captures the key semantic information in videos - the spatial-temporal dynamics of objects throughout the videos. VideoOrion employs expert vision models to extract object dynamics through a detect-segment-track pipeline, encoding them into a set of object tokens by aggregating spatial-temporal object features. Our method addresses the persistent challenge in Video-LLMs of efficiently compressing high-dimensional video data into semantic tokens that are comprehensible to LLMs. Compared to prior methods which resort to downsampling the original video or aggregating visual tokens using resamplers, leading to information loss and entangled semantics, VideoOrion not only offers a more natural and efficient way to derive compact, disentangled semantic representations but also enables explicit object modeling of video content with minimal computational cost. Moreover, the introduced object tokens naturally allow VideoOrion to accomplish video-based referring tasks. Experimental results show that VideoOrion can learn to make good use of the object tokens, and achieves competitive results on both general video question answering and video-based referring benchmarks.