PovNet+: A Deep Learning Architecture for Socially Assistive Robots to Learn and Assist with Multiple Activities of Daily Living

TL;DR

POVNet+ is a novel deep learning architecture enabling socially assistive robots to recognize multiple activities of daily living using multimodal data, thereby improving their ability to assist users proactively in real-world scenarios.

Contribution

This paper introduces POVNet+, the first multimodal deep learning model for multi-activity recognition in socially assistive robots, incorporating ADL and motion embeddings for better detection of known, new, and atypical activities.

Findings

POVNet+ outperforms state-of-the-art activity recognition methods in accuracy.

The architecture successfully identifies seen, unseen, and atypical ADLs in real environments.

Robots using POVNet+ can proactively initiate appropriate assistive behaviors.

Abstract

A significant barrier to the long-term deployment of autonomous socially assistive robots is their inability to both perceive and assist with multiple activities of daily living (ADLs). In this paper, we present the first multimodal deep learning architecture, POVNet+, for multi-activity recognition for socially assistive robots to proactively initiate assistive behaviors. Our novel architecture introduces the use of both ADL and motion embedding spaces to uniquely distinguish between a known ADL being performed, a new unseen ADL, or a known ADL being performed atypically in order to assist people in real scenarios. Furthermore, we apply a novel user state estimation method to the motion embedding space to recognize new ADLs while monitoring user performance. This ADL perception information is used to proactively initiate robot assistive interactions. Comparison experiments with state-of-the-art human activity recognition methods show our POVNet+ method has higher ADL classification accuracy. Human-robot interaction experiments in a cluttered living environment with multiple users and the socially assistive robot Leia using POVNet+ demonstrate the ability of our multi-modal ADL architecture in successfully identifying different seen and unseen ADLs, and ADLs being performed atypically, while initiating appropriate assistive human-robot interactions.