Co-training Transformer with Videos and Images Improves Action Recognition

TL;DR

This paper introduces Co-training Videos and Images for Action Recognition (CoVeR), a training paradigm that enhances video transformer models by jointly training on diverse video datasets and images, leading to improved action recognition accuracy.

Contribution

The paper proposes a novel co-training approach for video transformers that leverages both videos and images, demonstrating significant accuracy improvements across multiple datasets.

Findings

CoVeR improves Top-1 accuracy on Kinetics-400 by 2.4%.

CoVeR achieves state-of-the-art results on several action recognition datasets.

Joint training on diverse datasets and images enhances video transformer representations.

Abstract

In learning action recognition, models are typically pre-trained on object recognition with images, such as ImageNet, and later fine-tuned on target action recognition with videos. This approach has achieved good empirical performance especially with recent transformer-based video architectures. While recently many works aim to design more advanced transformer architectures for action recognition, less effort has been made on how to train video transformers. In this work, we explore several training paradigms and present two findings. First, video transformers benefit from joint training on diverse video datasets and label spaces (e.g., Kinetics is appearance-focused while SomethingSomething is motion-focused). Second, by further co-training with images (as single-frame videos), the video transformers learn even better video representations. We term this approach as Co-training Videos and Images for Action Recognition (CoVeR). In particular, when pretrained on ImageNet-21K based on the TimeSFormer architecture, CoVeR improves Kinetics-400 Top-1 Accuracy by 2.4%, Kinetics-600 by 2.3%, and SomethingSomething-v2 by 2.3%. When pretrained on larger-scale image datasets following previous state-of-the-art, CoVeR achieves best results on Kinetics-400 (87.2%), Kinetics-600 (87.9%), Kinetics-700 (79.8%), SomethingSomething-v2 (70.9%), and Moments-in-Time (46.1%), with a simple spatio-temporal video transformer.