PAL: Intelligence Augmentation using Egocentric Visual Context Detection

TL;DR

This paper introduces PAL, a wearable system for egocentric visual context detection that supports personalized, privacy-preserving intelligence augmentation through on-device deep learning and user-friendly interfaces.

Contribution

The paper presents a novel wearable system, PAL, integrating on-device deep learning for personalized context detection and clustering, advancing privacy-preserving intelligence augmentation.

Findings

Models achieved over 80% accuracy in in-the-wild contexts.

PAL effectively supports behavior change applications.

Open-source release promotes further research.

Abstract

Egocentric visual context detection can support intelligence augmentation applications. We created a wearable system, called PAL, for wearable, personalized, and privacy-preserving egocentric visual context detection. PAL has a wearable device with a camera, heart-rate sensor, on-device deep learning, and audio input/output. PAL also has a mobile/web application for personalized context labeling. We used on-device deep learning models for generic object and face detection, low-shot custom face and context recognition (e.g., activities like brushing teeth), and custom context clustering (e.g., indoor locations). The models had over 80\% accuracy in in-the-wild contexts (~1000 images) and we tested PAL for intelligence augmentation applications like behavior change. We have made PAL is open-source to further support intelligence augmentation using personalized and privacy-preserving egocentric visual contexts.