Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactions

TL;DR

This paper introduces MToMnet, a neural network model that explicitly encodes Theory of Mind to predict human beliefs and their changes during social interactions using multimodal data, advancing belief modeling in AI.

Contribution

The paper presents MToMnet, a novel neural network architecture that explicitly models Theory of Mind for belief prediction from multimodal inputs, outperforming existing methods with fewer parameters.

Findings

MToMnet significantly outperforms existing belief prediction methods.

MToMnet requires fewer parameters than comparable models.

The approach effectively predicts belief dynamics in real-world social interactions.

Abstract

We propose MToMnet - a Theory of Mind (ToM) neural network for predicting beliefs and their dynamics during human social interactions from multimodal input. ToM is key for effective nonverbal human communication and collaboration, yet, existing methods for belief modelling have not included explicit ToM modelling or have typically been limited to one or two modalities. MToMnet encodes contextual cues (scene videos and object locations) and integrates them with person-specific cues (human gaze and body language) in a separate MindNet for each person. Inspired by prior research on social cognition and computational ToM, we propose three different MToMnet variants: two involving fusion of latent representations and one involving re-ranking of classification scores. We evaluate our approach on two challenging real-world datasets, one focusing on belief prediction, while the other examining belief dynamics prediction. Our results demonstrate that MToMnet surpasses existing methods by a large margin while at the same time requiring a significantly smaller number of parameters. Taken together, our method opens up a highly promising direction for future work on artificial intelligent systems that can robustly predict human beliefs from their non-verbal behaviour and, as such, more effectively collaborate with humans.