Towards social pattern characterization in egocentric photo-streams

TL;DR

This paper proposes a comprehensive framework for analyzing social interactions and patterns in egocentric photo-streams using visual cues, event classification, and recurrence analysis, demonstrated on data from nine users.

Contribution

It introduces a novel multi-step approach combining social signal detection, event categorization with LSTM, and social pattern inference from egocentric images.

Findings

Promising results in social pattern characterization

Effective detection and categorization of social events

Insights into social relation diversity and frequency

Abstract

Following the increasingly popular trend of social interaction analysis in egocentric vision, this manuscript presents a comprehensive study for automatic social pattern characterization of a wearable photo-camera user, by relying on the visual analysis of egocentric photo-streams. The proposed framework consists of three major steps. The first step is to detect social interactions of the user where the impact of several social signals on the task is explored. The detected social events are inspected in the second step for categorization into different social meetings. These two steps act at event-level where each potential social event is modeled as a multi-dimensional time-series, whose dimensions correspond to a set of relevant features for each task, and LSTM is employed to classify the time-series. The last step of the framework is to characterize social patterns, which is essentially to infer the diversity and frequency of the social relations of the user through discovery of recurrences of the same people across the whole set of social events of the user. Experimental evaluation over a dataset acquired by 9 users demonstrates promising results on the task of social pattern characterization from egocentric photo-streams.