ProtoGAN: Towards Few Shot Learning for Action Recognition

TL;DR

ProtoGAN introduces a generative framework that synthesizes training examples for novel action categories in few-shot and generalized few-shot learning, significantly improving recognition accuracy.

Contribution

The paper proposes ProtoGAN, a novel generative approach using class prototypes to synthesize data for unseen classes, advancing few-shot and generalized few-shot action recognition.

Findings

Outperforms state-of-the-art in FSL on three datasets.

First to report results for G-FSL in action recognition.

Provides a new benchmark for future G-FSL research.

Abstract

Few-shot learning (FSL) for action recognition is a challenging task of recognizing novel action categories which are represented by few instances in the training data. In a more generalized FSL setting (G-FSL), both seen as well as novel action categories need to be recognized. Conventional classifiers suffer due to inadequate data in FSL setting and inherent bias towards seen action categories in G-FSL setting. In this paper, we address this problem by proposing a novel ProtoGAN framework which synthesizes additional examples for novel categories by conditioning a conditional generative adversarial network with class prototype vectors. These class prototype vectors are learnt using a Class Prototype Transfer Network (CPTN) from examples of seen categories. Our synthesized examples for a novel class are semantically similar to real examples belonging to that class and is used to train a model exhibiting better generalization towards novel classes. We support our claim by performing extensive experiments on three datasets: UCF101, HMDB51 and Olympic-Sports. To the best of our knowledge, we are the first to report the results for G-FSL and provide a strong benchmark for future research. We also outperform the state-of-the-art method in FSL for all the aforementioned datasets.