Federated Self-supervised Learning for Video Understanding

TL;DR

This paper introduces FedVSSL, a federated self-supervised learning framework for videos that addresses privacy and communication challenges, outperforming centralized methods on key video retrieval benchmarks.

Contribution

The paper proposes a novel federated SSL framework for videos, FedVSSL, with new aggregation strategies and partial weight updates, improving performance over centralized methods.

Findings

FedVSSL outperforms centralized SOTA by 6.66% on UCF-101

FedVSSL outperforms centralized SOTA by 5.13% on HMDB-51

Effective federated SSL for large-scale video data

Abstract

The ubiquity of camera-enabled mobile devices has lead to large amounts of unlabelled video data being produced at the edge. Although various self-supervised learning (SSL) methods have been proposed to harvest their latent spatio-temporal representations for task-specific training, practical challenges including privacy concerns and communication costs prevent SSL from being deployed at large scales. To mitigate these issues, we propose the use of Federated Learning (FL) to the task of video SSL. In this work, we evaluate the performance of current state-of-the-art (SOTA) video-SSL techniques and identify their shortcomings when integrated into the large-scale FL setting simulated with kinetics-400 dataset. We follow by proposing a novel federated SSL framework for video, dubbed FedVSSL, that integrates different aggregation strategies and partial weight updating. Extensive experiments demonstrate the effectiveness and significance of FedVSSL as it outperforms the centralized SOTA for the downstream retrieval task by 6.66% on UCF-101 and 5.13% on HMDB-51.