Learning Human Action Recognition Representations Without Real Humans

TL;DR

This paper explores pre-training human action recognition models without using real human images by utilizing human-removed real videos and synthetic data, achieving competitive performance while addressing privacy concerns.

Contribution

It introduces a novel benchmark and a pre-training strategy that combines synthetic and human-removed data, advancing privacy-preserving action recognition.

Findings

Outperforms previous methods by up to 5% on downstream tasks.

Closes the performance gap between human and no-human data representations.

Demonstrates effective transferability of privacy-preserving pre-trained models.

Abstract

Pre-training on massive video datasets has become essential to achieve high action recognition performance on smaller downstream datasets. However, most large-scale video datasets contain images of people and hence are accompanied with issues related to privacy, ethics, and data protection, often preventing them from being publicly shared for reproducible research. Existing work has attempted to alleviate these problems by blurring faces, downsampling videos, or training on synthetic data. On the other hand, analysis on the transferability of privacy-preserving pre-trained models to downstream tasks has been limited. In this work, we study this problem by first asking the question: can we pre-train models for human action recognition with data that does not include real humans? To this end, we present, for the first time, a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model. We then evaluate the transferability of the representation learned on this data to a diverse set of downstream action recognition benchmarks. Furthermore, we propose a novel pre-training strategy, called Privacy-Preserving MAE-Align, to effectively combine synthetic data and human-removed real data. Our approach outperforms previous baselines by up to 5% and closes the performance gap between human and no-human action recognition representations on downstream tasks, for both linear probing and fine-tuning. Our benchmark, code, and models are available at https://github.com/howardzh01/PPMA .