DynImg: Key Frames with Visual Prompts are Good Representation for Multi-Modal Video Understanding

TL;DR

This paper introduces DynImg, a novel video representation method using visual prompts from non-key frames to improve multi-modal video understanding by emphasizing fast-moving objects and maintaining spatio-temporal order.

Contribution

The paper proposes DynImg, a new approach that uses temporal prompts and 4D rotary position embedding to better capture fast-moving objects and spatio-temporal relationships in videos.

Findings

DynImg outperforms state-of-the-art methods by ~2% on multiple benchmarks.

Temporal prompts enhance the focus on fast-moving objects.

Maintaining spatio-temporal order improves understanding accuracy.

Abstract

In recent years, the introduction of Multi-modal Large Language Models (MLLMs) into video understanding tasks has become increasingly prevalent. However, how to effectively integrate temporal information remains a critical research focus. Traditional approaches treat spatial and temporal information separately. Due to issues like motion blur, it is challenging to accurately represent the spatial information of rapidly moving objects. This can lead to temporally important regions being underemphasized during spatial feature extraction, which in turn hinders accurate spatio-temporal interaction and video understanding. To address this limitation, we propose an innovative video representation method called Dynamic-Image (DynImg). Specifically, we introduce a set of non-key frames as temporal prompts to highlight the spatial areas containing fast-moving objects. During the process of visual feature extraction, these prompts guide the model to pay additional attention to the fine-grained spatial features corresponding to these regions. Moreover, to maintain the correct sequence for DynImg, we employ a corresponding 4D video Rotary Position Embedding. This retains both the temporal and spatial adjacency of DynImg, helping MLLM understand the spatio-temporal order within this combined format. Experimental evaluations reveal that DynImg surpasses the state-of-the-art methods by approximately 2% across multiple video understanding benchmarks, proving the effectiveness of our temporal prompts in enhancing video comprehension.