Using Features at Multiple Temporal and Spatial Resolutions to Predict Human Behavior in Real Time

TL;DR

This paper introduces a neural network-based approach that combines multi-resolution spatial and temporal features to predict human behavior in real time, enhancing accuracy in simulated USAR tasks.

Contribution

It presents a novel multi-resolution feature integration method for real-time human behavior prediction, trained jointly on high and low-resolution data.

Findings

Significant improvement in prediction accuracy over single-resolution methods.

Effective encoding of dynamic goals through high-resolution features.

Robust long-term behavior prediction using combined features.

Abstract

When performing complex tasks, humans naturally reason at multiple temporal and spatial resolutions simultaneously. We contend that for an artificially intelligent agent to effectively model human teammates, i.e., demonstrate computational theory of mind (ToM), it should do the same. In this paper, we present an approach for integrating high and low-resolution spatial and temporal information to predict human behavior in real time and evaluate it on data collected from human subjects performing simulated urban search and rescue (USAR) missions in a Minecraft-based environment. Our model composes neural networks for high and low-resolution feature extraction with a neural network for behavior prediction, with all three networks trained simultaneously. The high-resolution extractor encodes dynamically changing goals robustly by taking as input the Manhattan distance difference between the humans' Minecraft avatars and candidate goals in the environment for the latest few actions, computed from a high-resolution gridworld representation. In contrast, the low-resolution extractor encodes participants' historical behavior using a historical state matrix computed from a low-resolution graph representation. Through supervised learning, our model acquires a robust prior for human behavior prediction, and can effectively deal with long-term observations. Our experimental results demonstrate that our method significantly improves prediction accuracy compared to approaches that only use high-resolution information.