Self-supervised Multi-actor Social Activity Understanding in Streaming Videos

TL;DR

This paper introduces a self-supervised multi-actor social activity recognition method for streaming videos that models social interactions with minimal labeled data, outperforming traditional approaches in real-world scenarios.

Contribution

It proposes a novel self-supervised framework using a visual-semantic graph for modeling social interactions in streaming videos, reducing reliance on annotated data.

Findings

Achieves competitive results on standard group activity benchmarks.

Demonstrates strong generalization across multiple action localization datasets.

Effectively models social interactions with minimal supervision.

Abstract

This work addresses the problem of Social Activity Recognition (SAR), a critical component in real-world tasks like surveillance and assistive robotics. Unlike traditional event understanding approaches, SAR necessitates modeling individual actors' appearance and motions and contextualizing them within their social interactions. Traditional action localization methods fall short due to their single-actor, single-action assumption. Previous SAR research has relied heavily on densely annotated data, but privacy concerns limit their applicability in real-world settings. In this work, we propose a self-supervised approach based on multi-actor predictive learning for SAR in streaming videos. Using a visual-semantic graph structure, we model social interactions, enabling relational reasoning for robust performance with minimal labeled data. The proposed framework achieves competitive performance on standard group activity recognition benchmarks. Evaluation on three publicly available action localization benchmarks demonstrates its generalizability to arbitrary action localization.