MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models

TL;DR

MotionBench is a new benchmark designed to evaluate and improve the ability of vision language models to understand fine-grained motion in videos, revealing current limitations and proposing methods for enhancement.

Contribution

The paper introduces MotionBench, a comprehensive benchmark for fine-grained motion understanding, and proposes a novel Through-Encoder Fusion method to improve VLMs' motion perception.

Findings

Existing VLMs perform poorly in fine-grained motion understanding.

Higher frame rates and TE Fusion improve motion perception.

There is significant room for improvement in current models.

Abstract

In recent years, vision language models (VLMs) have made significant advancements in video understanding. However, a crucial capability - fine-grained motion comprehension - remains under-explored in current benchmarks. To address this gap, we propose MotionBench, a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. MotionBench evaluates models' motion-level perception through six primary categories of motion-oriented question types and includes data collected from diverse sources, ensuring a broad representation of real-world video content. Experimental results reveal that existing VLMs perform poorly in understanding fine-grained motions. To enhance VLM's ability to perceive fine-grained motion within a limited sequence length of LLM, we conduct extensive experiments reviewing VLM architectures optimized for video feature compression and propose a novel and efficient Through-Encoder (TE) Fusion method. Experiments show that higher frame rate inputs and TE Fusion yield improvements in motion understanding, yet there is still substantial room for enhancement. Our benchmark aims to guide and motivate the development of more capable video understanding models, emphasizing the importance of fine-grained motion comprehension. Project page: https://motion-bench.github.io .