Understanding Human Actions through the Lens of Executable Models

TL;DR

This paper introduces EXACT, a domain-specific language for representing human actions as executable motion programs, enabling better understanding, segmentation, and anomaly detection of human actions through neuro-symbolic models.

Contribution

The paper presents EXACT, a novel language for modeling human actions as executable programs, enhancing interpretability and data efficiency in action analysis.

Findings

Executable models improve data efficiency in action recognition.

Combining policies with program structure captures action relationships.

The approach enhances action segmentation and anomaly detection.

Abstract

Human-centred systems require an understanding of human actions in the physical world. Temporally extended sequences of actions are intentional and structured, yet existing methods for recognising what actions are performed often do not attempt to capture their structure, particularly how the actions are executed. This, however, is crucial for assessing the quality of the action's execution and its differences from other actions. To capture the internal mechanics of actions, we introduce a domain-specific language EXACT that represents human motions as underspecified motion programs, interpreted as reward-generating functions for zero-shot policy inference using forward-backwards representations. By leveraging the compositional nature of EXACT motion programs, we combine individual policies into an executable neuro-symbolic model that uses program structure for compositional modelling. We evaluate the utility of the proposed pipeline for creating executable action models by analysing motion-capture data to understand human actions, for the tasks of human action segmentation and action anomaly detection. Our results show that the use of executable action models improves data efficiency and captures intuitive relationships between actions compared with monolithic, task-specific approaches.