Detect, anticipate and generate: Semi-supervised recurrent latent variable models for human activity modeling

TL;DR

This paper introduces semi-supervised variational recurrent neural networks that model human activity sequences, predict future actions, and incorporate discrete labels, improving activity detection and anticipation in human-robot collaboration.

Contribution

The work presents a novel semi-supervised recurrent latent variable model capable of modeling latent factors, integrating labels, and generating future activity sequences.

Findings

Outperforms state-of-the-art in activity detection

Achieves better anticipation accuracy

Generates plausible future action sequences

Abstract

Successful Human-Robot collaboration requires a predictive model of human behavior. The robot needs to be able to recognize current goals and actions and to predict future activities in a given context. However, the spatio-temporal sequence of human actions is difficult to model since latent factors such as intention, task, knowledge, intuition and preference determine the action choices of each individual. In this work we introduce semi-supervised variational recurrent neural networks which are able to a) model temporal distributions over latent factors and the observable feature space, b) incorporate discrete labels such as activity type when available, and c) generate possible future action sequences on both feature and label level. We evaluate our model on the Cornell Activity Dataset CAD-120 dataset. Our model outperforms state-of-the-art approaches in both activity and affordance detection and anticipation. Additionally, we show how samples of possible future action sequences are in line with past observations.