
TL;DR
This paper investigates training-free token merging in video transformers, demonstrating it can significantly speed up processing by about 2.5 times with minimal accuracy loss across various datasets.
Contribution
It provides the first comprehensive evaluation of token merging in video transformers on complex datasets, establishing best practices for this technique.
Findings
Achieves around 2.5X speedup in video processing.
Maintains high accuracy with only -0.55% average loss.
Validates effectiveness across multiple datasets and models.
Abstract
Video transformer models require huge amounts of compute resources due to the spatio-temporal scaling of the input. Tackling this, recent methods have proposed to drop or merge tokens for image models, whether randomly or via learned methods. Merging tokens has many benefits: it can be plugged into any vision transformer, does not require model re-training, and it propagates information that would otherwise be dropped through the model. Before now, video token merging has not been evaluated on temporally complex datasets for video understanding. In this work, we explore training-free token merging for video to provide comprehensive experiments and find best practices across four video transformers on three datasets that exhibit coarse and fine-grained action recognition. Our results showcase the benefits of video token merging with a speedup of around X while maintaining accuracy…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
