Clustering Egocentric Images in Passive Dietary Monitoring with Self-Supervised Learning

TL;DR

This paper introduces a self-supervised learning framework to cluster egocentric images from wearable cameras, improving data annotation efficiency for dietary monitoring in low-resource settings.

Contribution

The novel self-supervised clustering method effectively organizes large volumes of egocentric images into meaningful events, aiding dietary assessment.

Findings

Outperforms baseline clustering methods in quality and accuracy

Validated on real-world dietary monitoring data from Ghana

Facilitates more efficient data annotation for dietary analysis

Abstract

In our recent dietary assessment field studies on passive dietary monitoring in Ghana, we have collected over 250k in-the-wild images. The dataset is an ongoing effort to facilitate accurate measurement of individual food and nutrient intake in low and middle income countries with passive monitoring camera technologies. The current dataset involves 20 households (74 subjects) from both the rural and urban regions of Ghana, and two different types of wearable cameras were used in the studies. Once initiated, wearable cameras continuously capture subjects' activities, which yield massive amounts of data to be cleaned and annotated before analysis is conducted. To ease the data post-processing and annotation tasks, we propose a novel self-supervised learning framework to cluster the large volume of egocentric images into separate events. Each event consists of a sequence of temporally continuous and contextually similar images. By clustering images into separate events, annotators and dietitians can examine and analyze the data more efficiently and facilitate the subsequent dietary assessment processes. Validated on a held-out test set with ground truth labels, the proposed framework outperforms baselines in terms of clustering quality and classification accuracy.