Inferring Past Human Actions in Homes with Abductive Reasoning

TL;DR

This paper introduces a new abductive reasoning task to infer past human actions in homes from a single image, proposing models that improve understanding of human behavior for applications like robotics and health monitoring.

Contribution

It presents the novel task of abductive past action inference and develops models, including the BiGED, that outperform existing methods in reasoning about past human actions.

Findings

BiGED outperforms other models on the Action Genome dataset.

Models effectively infer plausible past actions from current scene evidence.

Advances enable better understanding of human behaviors for real-world applications.

Abstract

Abductive reasoning aims to make the most likely inference for a given set of incomplete observations. In this paper, we introduce "Abductive Past Action Inference", a novel research task aimed at identifying the past actions performed by individuals within homes to reach specific states captured in a single image, using abductive inference. The research explores three key abductive inference problems: past action set prediction, past action sequence prediction, and abductive past action verification. We introduce several models tailored for abductive past action inference, including a relational graph neural network, a relational bilinear pooling model, and a relational transformer model. Notably, the newly proposed object-relational bilinear graph encoder-decoder (BiGED) model emerges as the most effective among all methods evaluated, demonstrating good proficiency in handling the intricacies of the Action Genome dataset. The contributions of this research significantly advance the ability of deep learning models to reason about current scene evidence and make highly plausible inferences about past human actions. This advancement enables a deeper understanding of events and behaviors, which can enhance decision-making and improve system capabilities across various real-world applications such as Human-Robot Interaction and Elderly Care and Health Monitoring. Code and data available at https://github.com/LUNAProject22/AAR