Video Token Merging for Long-form Video Understanding

TL;DR

This paper introduces a learnable video token merging method that enhances long-form video understanding by reducing memory usage and increasing processing speed while maintaining high accuracy.

Contribution

It proposes a novel saliency-aware, learnable token merging algorithm for long-form videos, improving efficiency and performance over existing methods.

Findings

Achieves comparable or better accuracy on multiple datasets.

Reduces memory costs by 84%.

Increases throughput by approximately 6.89 times.

Abstract

As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.