Anticipating Daily Intention using On-Wrist Motion Triggered Sensing

TL;DR

This paper introduces an on-wrist motion sensing system utilizing a novel RNN and Policy Network to anticipate daily human intentions with high accuracy while reducing visual observation processing.

Contribution

The paper presents a new on-wrist sensing approach with a combined RNN and Policy Network for intention prediction, trained jointly with policy gradient and cross-entropy loss.

Findings

Achieved over 90% accuracy in intention prediction.

Processed only 29% of visual data on average.

Collected the first daily intention dataset with 2379 videos.

Abstract

Anticipating human intention by observing one's actions has many applications. For instance, picking up a cellphone, then a charger (actions) implies that one wants to charge the cellphone (intention). By anticipating the intention, an intelligent system can guide the user to the closest power outlet. We propose an on-wrist motion triggered sensing system for anticipating daily intentions, where the on-wrist sensors help us to persistently observe one's actions. The core of the system is a novel Recurrent Neural Network (RNN) and Policy Network (PN), where the RNN encodes visual and motion observation to anticipate intention, and the PN parsimoniously triggers the process of visual observation to reduce computation requirement. We jointly trained the whole network using policy gradient and cross-entropy loss. To evaluate, we collect the first daily "intention" dataset consisting of 2379 videos with 34 intentions and 164 unique action sequences. Our method achieves 92.68%, 90.85%, 97.56% accuracy on three users while processing only 29% of the visual observation on average.