SEAL: Semantic Attention Learning for Long Video Representation

TL;DR

SEAL introduces a semantic attention learning framework that efficiently represents long videos by decomposing them into scenes, objects, and actions, reducing redundancy and computational cost for improved performance in various understanding tasks.

Contribution

The paper proposes a novel unified long video representation method using semantic entities and an attention module that balances relevance and diversity, advancing long video understanding.

Findings

Outperforms state-of-the-art in video question answering

Achieves superior results in temporal grounding tasks

Demonstrates versatility across multiple benchmarks

Abstract

Long video understanding presents challenges due to the inherent high computational complexity and redundant temporal information. An effective representation for long videos must efficiently process such redundancy while preserving essential contents for downstream tasks. This paper introduces SEmantic Attention Learning (SEAL), a novel unified representation for long videos. To reduce computational complexity, long videos are decomposed into three distinct types of semantic entities: scenes, objects, and actions, allowing models to operate on a compact set of entities rather than a large number of frames or pixels. To further address redundancy, we propose an attention learning module that balances token relevance with diversity, formulated as a subset selection optimization problem. Our representation is versatile and applicable across various long video understanding tasks. Extensive experiments demonstrate that SEAL significantly outperforms state-of-the-art methods in video question answering and temporal grounding tasks across diverse benchmarks, including LVBench, MovieChat-1K, and Ego4D.