ProActive: Self-Attentive Temporal Point Process Flows for Activity Sequences

TL;DR

ProActive introduces a self-attentive neural framework using temporal point processes to model, predict, and generate human activity sequences in continuous time, addressing challenges like goal prediction and sequence generation.

Contribution

It presents a novel neural marked temporal point process model with self-attention and normalizing flows for activity sequence analysis, including goal detection and sequence generation.

Findings

Significant accuracy improvements over state-of-the-art in action and goal prediction.

First application of end-to-end activity sequence generation.

Effective early goal detection with limited actions.

Abstract

Any human activity can be represented as a temporal sequence of actions performed to achieve a certain goal. Unlike machine-made time series, these action sequences are highly disparate as the time taken to finish a similar action might vary between different persons. Therefore, understanding the dynamics of these sequences is essential for many downstream tasks such as activity length prediction, goal prediction, etc. Existing neural approaches that model an activity sequence are either limited to visual data or are task specific, i.e., limited to next action or goal prediction. In this paper, we present ProActive, a neural marked temporal point process (MTPP) framework for modeling the continuous-time distribution of actions in an activity sequence while simultaneously addressing three high-impact problems -- next action prediction, sequence-goal prediction, and end-to-end sequence generation. Specifically, we utilize a self-attention module with temporal normalizing flows to model the influence and the inter-arrival times between actions in a sequence. Moreover, for time-sensitive prediction, we perform an early detection of sequence goal via a constrained margin-based optimization procedure. This in-turn allows ProActive to predict the sequence goal using a limited number of actions. Extensive experiments on sequences derived from three activity recognition datasets show the significant accuracy boost of ProActive over the state-of-the-art in terms of action and goal prediction, and the first-ever application of end-to-end action sequence generation.