ELSA: Evaluating Localization of Social Activities in Urban Streets using Open-Vocabulary Detection

TL;DR

ELSA introduces a comprehensive benchmark for evaluating social activity localization in urban street images, highlighting the limitations of current open-vocabulary detection models in multi-label, multi-activity scenarios.

Contribution

This work provides the first multi-label social activity detection benchmark with over 900 annotated images and introduces novel confidence and aggregation methods for better evaluation.

Findings

Current models struggle with semantic consistency in social activity detection.

Existing models often produce overconfident predictions that lack contextual accuracy.

ELSA reveals significant limitations in state-of-the-art models for multi-label social activity localization.

Abstract

Existing Open Vocabulary Detection (OVD) models exhibit a number of challenges. They often struggle with semantic consistency across diverse inputs, and are often sensitive to slight variations in input phrasing, leading to inconsistent performance. The calibration of their predictive confidence, especially in complex multi-label scenarios, remains suboptimal, frequently resulting in overconfident predictions that do not accurately reflect their context understanding. To understand these limitations, multi-label detection benchmarks are needed. A particularly challenging domain for such benchmarking is social activities. Due to the lack of multi-label benchmarks for social interactions, in this work we present ELSA: Evaluating Localization of Social Activities. ELSA draws on theoretical frameworks in urban sociology and design and uses in-the-wild street-level imagery, where the size of groups and the types of activities vary significantly. ELSA includes more than 900 manually annotated images with more than 4,300 multi-labeled bounding boxes for individual and group activities. We introduce a novel confidence score computation method NLSE and a novel Dynamic Box Aggregation (DBA) algorithm to assess semantic consistency in overlapping predictions. We report our results on the widely-used SOTA models Grounding DINO, Detic, OWL, and MDETR. Our evaluation protocol considers semantic stability and localization accuracy and further exposes the limitations of existing approaches.