Towards Long Video Understanding via Fine-detailed Video Story Generation

TL;DR

This paper introduces FDVS, a method that converts long videos into detailed hierarchical textual descriptions, improving understanding by modeling fine-grained content and reducing redundancy, applicable across multiple tasks without fine-tuning.

Contribution

The paper proposes a novel hierarchical textual representation approach with a bottom-up interpretation mechanism and redundancy reduction, enabling versatile long video understanding.

Findings

Effective across eight datasets and three tasks.

Improves long-context relationship modeling.

Reduces redundancy at visual and textual levels.

Abstract

Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method.