EgoGroups: A Benchmark For Detecting Social Groups of People in the Wild

TL;DR

EgoGroups introduces a diverse first-person dataset capturing social interactions worldwide, enabling evaluation of models in real-world, unconstrained environments and revealing cultural and crowd density effects on social group detection.

Contribution

The paper presents EgoGroups, a novel first-person social group dataset with extensive annotations across diverse global settings, addressing limitations of prior third-person datasets.

Findings

VLMs and LLMs outperform supervised models in zero-shot group detection.

Crowd density impacts model performance.

Cultural regions influence social group detection accuracy.

Abstract

Social group detection, or the identification of humans involved in reciprocal interpersonal interactions (e.g., family members, friends, and customers and merchants), is a crucial component of social intelligence needed for agents transacting in the world. The few existing benchmarks for social group detection are limited by low scene diversity and reliance on third-person camera sources (e.g., surveillance footage). Consequently, these benchmarks generally lack real-world evaluation on how groups form and evolve in diverse cultural contexts and unconstrained settings. To address this gap, we introduce EgoGroups, a first-person view dataset that captures social dynamics in cities around the world. EgoGroups spans 65 countries covering low, medium, and high-crowd settings under four weather/time-of-day conditions. We include dense human annotations for person and social groups, along with rich geographic and scene metadata. Using this dataset, we performed an extensive evaluation of state-of-the-art VLM/LLMs and supervised models on their group detection capabilities. We found several interesting findings, including VLMs and LLMs can outperform supervised baselines in a zero-shot setting, while crowd density and cultural regions clearly influence model performance.