Grounding Social Perception in Intuitive Physics

TL;DR

This paper introduces a physics-grounded Bayesian model, SIMPLE, that infers social agents' goals and relationships from animations, aligning closely with human judgments and emphasizing the role of intuitive physics in social perception.

Contribution

The paper presents a novel computational model, SIMPLE, integrating physics simulation and inverse planning to understand social interactions, validated on a new dataset of procedurally generated animations.

Findings

SIMPLE achieved high accuracy in inferring goals and relations from animations.

SIMPLE's performance closely matched human judgments across diverse scenarios.

Physics-agnostic models failed to replicate human-like social scene understanding.

Abstract

People infer rich social information from others' actions. These inferences are often constrained by the physical world: what agents can do, what obstacles permit, and how the physical actions of agents causally change an environment and other agents' mental states and behavior. We propose that such rich social perception is more than visual pattern matching, but rather a reasoning process grounded in an integration of intuitive psychology with intuitive physics. To test this hypothesis, we introduced PHASE (PHysically grounded Abstract Social Events), a large dataset of procedurally generated animations, depicting physically simulated two-agent interactions on a 2D surface. Each animation follows the style of the Heider and Simmel movie, with systematic variation in environment geometry, object dynamics, agent capacities, goals, and relationships (friendly/adversarial/neutral). We then present a computational model, SIMPLE, a physics-grounded Bayesian inverse planning model that integrates planning, probabilistic planning, and physics simulation to infer agents' goals and relations from their trajectories. Our experimental results showed that SIMPLE achieved high accuracy and agreement with human judgments across diverse scenarios, while feedforward baseline models -- including strong vision-language models -- and physics-agnostic inverse planning failed to achieve human-level performance and did not align with human judgments. These results suggest that our model provides a computational account for how people understand physically grounded social scenes by inverting a generative model of physics and agents.