Exploiting Motion Information from Unlabeled Videos for Static Image Action Recognition

TL;DR

This paper introduces a unified framework that leverages unlabeled videos to improve static image action recognition by enhancing visual features and augmenting motion information, reducing reliance on labeled data.

Contribution

It combines self-supervised learning and auxiliary motion representation generation into a single framework for better image action recognition.

Findings

Outperforms existing methods on four benchmark datasets.

Effectively utilizes unlabeled videos to improve recognition accuracy.

Reduces dependence on large labeled datasets.

Abstract

Static image action recognition, which aims to recognize action based on a single image, usually relies on expensive human labeling effort such as adequate labeled action images and large-scale labeled image dataset. In contrast, abundant unlabeled videos can be economically obtained. Therefore, several works have explored using unlabeled videos to facilitate image action recognition, which can be categorized into the following two groups: (a) enhance visual representations of action images with a designed proxy task on unlabeled videos, which falls into the scope of self-supervised learning; (b) generate auxiliary representations for action images with the generator learned from unlabeled videos. In this paper, we integrate the above two strategies in a unified framework, which consists of Visual Representation Enhancement (VRE) module and Motion Representation Augmentation (MRA) module. Specifically, the VRE module includes a proxy task which imposes pseudo motion label constraint and temporal coherence constraint on unlabeled videos, while the MRA module could predict the motion information of a static action image by exploiting unlabeled videos. We demonstrate the superiority of our framework based on four benchmark human action datasets with limited labeled data.