STPrivacy: Spatio-Temporal Privacy-Preserving Action Recognition

TL;DR

STPrivacy introduces a novel spatio-temporal framework using vision Transformers for privacy-preserving action recognition, effectively removing privacy information while maintaining recognition accuracy.

Contribution

It is the first to incorporate vision Transformers into video-level privacy-preserving action recognition and proposes mechanisms for spatio-temporal privacy removal.

Findings

Effective privacy removal demonstrated on new benchmarks VP-HMDB51 and VP-UCF101.

Improved privacy-accuracy trade-off compared to existing methods.

Framework generalizes well across different datasets and tasks.

Abstract

Existing methods of privacy-preserving action recognition (PPAR) mainly focus on frame-level (spatial) privacy removal through 2D CNNs. Unfortunately, they have two major drawbacks. First, they may compromise temporal dynamics in input videos, which are critical for accurate action recognition. Second, they are vulnerable to practical attacking scenarios where attackers probe for privacy from an entire video rather than individual frames. To address these issues, we propose a novel framework STPrivacy to perform video-level PPAR. For the first time, we introduce vision Transformers into PPAR by treating a video as a tubelet sequence, and accordingly design two complementary mechanisms, i.e., sparsification and anonymization, to remove privacy from a spatio-temporal perspective. In specific, our privacy sparsification mechanism applies adaptive token selection to abandon action-irrelevant tubelets. Then, our anonymization mechanism implicitly manipulates the remaining action-tubelets to erase privacy in the embedding space through adversarial learning. These mechanisms provide significant advantages in terms of privacy preservation for human eyes and action-privacy trade-off adjustment during deployment. We additionally contribute the first two large-scale PPAR benchmarks, VP-HMDB51 and VP-UCF101, to the community. Extensive evaluations on them, as well as two other tasks, validate the effectiveness and generalization capability of our framework.