HERMES: temporal-coHERent long-forM understanding with Episodes and Semantics

TL;DR

HERMES is a novel framework for long-form video understanding that combines efficient episodic compression and semantic retrieval to improve accuracy and reduce computational costs, outperforming existing models.

Contribution

This paper introduces HERMES, a new approach with two modules that enhance long-video understanding by efficiently capturing temporal and semantic information, adaptable as a standalone or integrated system.

Findings

Reduces inference latency by up to 43%

Decreases memory usage by 46%

Achieves state-of-the-art results on multiple benchmarks

Abstract

Long-form video understanding presents unique challenges that extend beyond traditional short-video analysis approaches, particularly in capturing long-range dependencies, processing redundant information efficiently, and extracting high-level semantic concepts. To address these challenges, we propose a novel approach that more accurately reflects human cognition. This paper introduces HERMES: temporal-coHERent long-forM understanding with Episodes and Semantics, featuring two versatile modules that can enhance existing video-language models or operate as a standalone system. Our Episodic COmpressor (ECO) efficiently aggregates representations from micro to semi-macro levels, reducing computational overhead while preserving temporal dependencies. Our Semantics ReTRiever (SeTR) enriches these representations with semantic information by focusing on broader context, dramatically reducing feature dimensionality while preserving relevant macro-level information. We demonstrate that these modules can be seamlessly integrated into existing SOTA models, consistently improving their performance while reducing inference latency by up to 43% and memory usage by 46%. As a standalone system, HERMES achieves state-of-the-art performance across multiple long-video understanding benchmarks in both zero-shot and fully-supervised settings.