Joint learning of images and videos with a single Vision Transformer

TL;DR

This paper introduces a unified Vision Transformer model that jointly learns from images and videos, leveraging temporal aggregation for videos, and demonstrates its effectiveness across multiple datasets.

Contribution

The paper presents a novel single-model approach for joint image and video learning using Vision Transformer with temporal aggregation.

Findings

Effective on multiple image and action recognition datasets

Achieves competitive performance with separate models

Demonstrates the feasibility of unified image-video learning

Abstract

In this study, we propose a method for jointly learning of images and videos using a single model. In general, images and videos are often trained by separate models. We propose in this paper a method that takes a batch of images as input to Vision Transformer IV-ViT, and also a set of video frames with temporal aggregation by late fusion. Experimental results on two image datasets and two action recognition datasets are presented.