PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatch

TL;DR

PrISM-Observer is a smartwatch-based system that monitors and supports users during daily procedures by providing real-time, proactive interventions to prevent errors, especially aiding those with cognitive challenges.

Contribution

This paper introduces a novel, real-time, multimodal sensing system that observes user actions and intervenes proactively during daily tasks, a significant advancement over traditional reactive systems.

Findings

Validated steps-tracking accuracy across three datasets

Demonstrated effective real-time interventions in a cooking scenario

Received positive user feedback on system usefulness

Abstract

We routinely perform procedures (such as cooking) that include a set of atomic steps. Often, inadvertent omission or misordering of a single step can lead to serious consequences, especially for those experiencing cognitive challenges such as dementia. This paper introduces PrISM-Observer, a smartwatch-based, context-aware, real-time intervention system designed to support daily tasks by preventing errors. Unlike traditional systems that require users to seek out information, the agent observes user actions and intervenes proactively. This capability is enabled by the agent's ability to continuously update its belief in the user's behavior in real-time through multimodal sensing and forecast optimal intervention moments and methods. We first validated the steps-tracking performance of our framework through evaluations across three datasets with different complexities. Then, we implemented a real-time agent system using a smartwatch and conducted a user study in a cooking task scenario. The system generated helpful interventions, and we gained positive feedback from the participants. The general applicability of PrISM-Observer to daily tasks promises broad applications, for instance, including support for users requiring more involved interventions, such as people with dementia or post-surgical patients.