PatchBlender: A Motion Prior for Video Transformers
Gabriele Prato, Yale Song, Janarthanan Rajendran, R Devon Hjelm, Neel, Joshi, Sarath Chandar

TL;DR
PatchBlender introduces a learnable, lightweight blending function for video transformers that effectively models temporal patterns, improving performance on video datasets while maintaining compatibility with various architectures.
Contribution
It proposes PatchBlender, a novel learnable temporal prior for video transformers that enhances temporal encoding without significant computational overhead.
Findings
Improves performance of video transformers on benchmark datasets.
Compatible with most Transformer architectures.
Extremely lightweight, using only 0.005% GFLOPs of ViT-B.
Abstract
Transformers have become one of the dominant architectures in the field of computer vision. However, there are yet several challenges when applying such architectures to video data. Most notably, these models struggle to model the temporal patterns of video data effectively. Directly targeting this issue, we introduce PatchBlender, a learnable blending function that operates over patch embeddings across the temporal dimension of the latent space. We show that our method is successful at enabling vision transformers to encode the temporal component of video data. On Something-Something v2 and MOVi-A, we show that our method improves the baseline performance of video Transformers. PatchBlender has the advantage of being compatible with almost any Transformer architecture and since it is learnable, the model can adaptively turn on or off the prior. It is also extremely lightweight…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Advanced Vision and Imaging · Human Pose and Action Recognition
MethodsMulti-Head Attention · Attention Is All You Need · Layer Normalization · Adam · Softmax · Dropout · Byte Pair Encoding · Position-Wise Feed-Forward Layer · Label Smoothing · Linear Layer
