On the Surprising Effectiveness of Transformers in Low-Labeled Video Recognition

TL;DR

This paper demonstrates that vision transformers outperform CNNs in low-labeled video classification, challenging prior assumptions about their data requirements and suggesting their broader applicability in semi-supervised settings.

Contribution

The study empirically shows that video transformers are highly effective in low-labeled data regimes, outperforming CNNs and semi-supervised methods, with thorough analysis and ablation studies.

Findings

Transformers perform well in low-labeled video classification.

Transformers outperform semi-supervised CNN methods on benchmark datasets.

Analysis explains the effectiveness of transformers in low-data regimes.

Abstract

Recently vision transformers have been shown to be competitive with convolution-based methods (CNNs) broadly across multiple vision tasks. The less restrictive inductive bias of transformers endows greater representational capacity in comparison with CNNs. However, in the image classification setting this flexibility comes with a trade-off with respect to sample efficiency, where transformers require ImageNet-scale training. This notion has carried over to video where transformers have not yet been explored for video classification in the low-labeled or semi-supervised settings. Our work empirically explores the low data regime for video classification and discovers that, surprisingly, transformers perform extremely well in the low-labeled video setting compared to CNNs. We specifically evaluate video vision transformers across two contrasting video datasets (Kinetics-400 and SomethingSomething-V2) and perform thorough analysis and ablation studies to explain this observation using the predominant features of video transformer architectures. We even show that using just the labeled data, transformers significantly outperform complex semi-supervised CNN methods that leverage large-scale unlabeled data as well. Our experiments inform our recommendation that semi-supervised learning video work should consider the use of video transformers in the future.