Diving Deep into the Motion Representation of Video-Text Models

TL;DR

This paper investigates the ability of video-text models to understand motion in videos, introduces GPT-4 generated motion descriptions for evaluation, and proposes a method to improve motion understanding in these models.

Contribution

It introduces a novel approach using GPT-4 generated motion descriptions and demonstrates a method to enhance motion understanding in video-text models.

Findings

Video-text models underperform compared to humans in motion understanding.

The proposed method improves motion description retrieval accuracy.

Highlighting the importance of high-quality, fine-grained motion captions in datasets.

Abstract

Videos are more informative than images because they capture the dynamics of the scene. By representing motion in videos, we can capture dynamic activities. In this work, we introduce GPT-4 generated motion descriptions that capture fine-grained motion descriptions of activities and apply them to three action datasets. We evaluated several video-text models on the task of retrieval of motion descriptions. We found that they fall far behind human expert performance on two action datasets, raising the question of whether video-text models understand motion in videos. To address it, we introduce a method of improving motion understanding in video-text models by utilizing motion descriptions. This method proves to be effective on two action datasets for the motion description retrieval task. The results draw attention to the need for quality captions involving fine-grained motion information in existing datasets and demonstrate the effectiveness of the proposed pipeline in understanding fine-grained motion during video-text retrieval.