Video, How Do Your Tokens Merge?

TL;DR

This paper investigates training-free token merging in video transformers, demonstrating it can significantly speed up processing by about 2.5 times with minimal accuracy loss across various datasets.

Contribution

It provides the first comprehensive evaluation of token merging in video transformers on complex datasets, establishing best practices for this technique.

Findings

Achieves around 2.5X speedup in video processing.

Maintains high accuracy with only -0.55% average loss.

Validates effectiveness across multiple datasets and models.

Abstract

Video transformer models require huge amounts of compute resources due to the spatio-temporal scaling of the input. Tackling this, recent methods have proposed to drop or merge tokens for image models, whether randomly or via learned methods. Merging tokens has many benefits: it can be plugged into any vision transformer, does not require model re-training, and it propagates information that would otherwise be dropped through the model. Before now, video token merging has not been evaluated on temporally complex datasets for video understanding. In this work, we explore training-free token merging for video to provide comprehensive experiments and find best practices across four video transformers on three datasets that exhibit coarse and fine-grained action recognition. Our results showcase the benefits of video token merging with a speedup of around $2.5$X while maintaining accuracy (avg. $-0.55\%$ for ViViT). Code available at https://github.com/sjpollard/video-how-do-your-tokens-merge.