ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices

TL;DR

ProMemAssist is a wearable AI system that models users' working memory in real-time to provide timely, context-sensitive assistance, improving user engagement and support in cognitively demanding tasks.

Contribution

This work introduces a novel WM-based model for proactive assistance in wearable devices, grounded in cognitive theories and utilizing multi-modal sensor data.

Findings

ProMemAssist delivers more selective assistance than baseline systems.

Participants showed higher engagement with ProMemAssist.

Qualitative feedback supports the benefits of WM modeling for nuanced support.

Abstract

Wearable AI systems aim to provide timely assistance in daily life, but existing approaches often rely on user initiation or predefined task knowledge, neglecting users' current mental states. We introduce ProMemAssist, a smart glasses system that models a user's working memory (WM) in real-time using multi-modal sensor signals. Grounded in cognitive theories of WM, our system represents perceived information as memory items and episodes with encoding mechanisms, such as displacement and interference. This WM model informs a timing predictor that balances the value of assistance with the cost of interruption. In a user study with 12 participants completing cognitively demanding tasks, ProMemAssist delivered more selective assistance and received higher engagement compared to an LLM baseline system. Qualitative feedback highlights the benefits of WM modeling for nuanced, context-sensitive support, offering design implications for more attentive and user-aware proactive agents.