End-to-End Semantic Video Transformer for Zero-Shot Action Recognition

TL;DR

This paper introduces an end-to-end transformer model for zero-shot video action recognition that captures long-range dependencies and outperforms existing methods on standard datasets, with a new setup ensuring proper zero-shot evaluation.

Contribution

It presents a novel transformer-based approach for zero-shot action recognition and a new experimental setup to properly evaluate zero-shot capabilities.

Findings

Outperforms state-of-the-art in zero-shot accuracy on UCF-101, HMDB-51, and ActivityNet

Efficiently captures long-range spatiotemporal dependencies

Provides a new standardized setup for zero-shot action recognition

Abstract

While video action recognition has been an active area of research for several years, zero-shot action recognition has only recently started gaining traction. In this work, we propose a novel end-to-end trained transformer model which is capable of capturing long range spatiotemporal dependencies efficiently, contrary to existing approaches which use 3D-CNNs. Moreover, to address a common ambiguity in the existing works about classes that can be considered as previously unseen, we propose a new experimentation setup that satisfies the zero-shot learning premise for action recognition by avoiding overlap between the training and testing classes. The proposed approach significantly outperforms the state of the arts in zero-shot action recognition in terms of the the top-1 accuracy on UCF-101, HMDB-51 and ActivityNet datasets. The code and proposed experimentation setup are available in GitHub: https://github.com/Secure-and-Intelligent-Systems-Lab/SemanticVideoTransformer