FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding

TL;DR

FLoC is a training-free, model-agnostic visual token compression method for long video understanding that efficiently selects representative tokens, significantly reducing computational load while maintaining high performance across various benchmarks.

Contribution

The paper introduces FLoC, a novel facility location-based token compression framework that is efficient, versatile, and guarantees near-optimal selection without additional training.

Findings

Outperforms recent compression techniques on multiple benchmarks.

Drastically reduces visual tokens while maintaining performance.

Efficiently integrates with diverse video-LMMs and workflows.

Abstract

Recent studies in long video understanding have harnessed the advanced visual-language reasoning capabilities of Large Multimodal Models (LMMs), driving the evolution of video-LMMs specialized for processing extended video sequences. However, the scalability of these models is severely limited by the overwhelming volume of visual tokens generated from extended video sequences. To address this challenge, we propose FLoC, an efficient visual token compression framework based on the facility location function, a principled approach that swiftly selects a compact yet highly representative and diverse subset of visual tokens within a predefined budget on the number of visual tokens. By integrating the lazy greedy algorithm, our method achieves remarkable efficiency gains by swiftly selecting a compact subset of tokens, drastically reducing the number of visual tokens while guaranteeing near-optimal performance. Notably, our approach is training-free, model-agnostic, and query-agnostic, providing a versatile solution that seamlessly integrates with diverse video-LLMs and existing workflows. Extensive evaluations on large-scale benchmarks, such as Video-MME, MLVU, LongVideoBench, and EgoSchema, show that our framework consistently surpasses recent compression techniques, highlighting its effectiveness and robustness in addressing the challenges of long video understanding as well as its processing efficiency.