S$^3$IT: A Benchmark for Spatially Situated Social Intelligence Test

TL;DR

This paper introduces S$^3$IT, a benchmark for evaluating embodied social intelligence in agents through a complex seat-ordering task involving physical and social reasoning in 3D environments.

Contribution

It presents a novel benchmark and framework for assessing embodied social intelligence, combining physical environment perception, social norm understanding, and multi-objective optimization.

Findings

State-of-the-art LLMs perform poorly on S$^3$IT

Humans significantly outperform LLMs in the task

LLMs show deficiencies in spatial reasoning but can resolve conflicts with explicit cues

Abstract

The integration of embodied agents into human environments demands embodied social intelligence: reasoning over both social norms and physical constraints. However, existing evaluations fail to address this integration, as they are limited to either disembodied social reasoning (e.g., in text) or socially-agnostic physical tasks. Both approaches fail to assess an agent's ability to integrate and trade off both physical and social constraints within a realistic, embodied context. To address this challenge, we introduce Spatially Situated Social Intelligence Test (S$^{3}$IT), a benchmark specifically designed to evaluate embodied social intelligence. It is centered on a novel and challenging seat-ordering task, requiring an agent to arrange seating in a 3D environment for a group of large language model-driven (LLM-driven) NPCs with diverse identities, preferences, and intricate interpersonal relationships. Our procedurally extensible framework generates a vast and diverse scenario space with controllable difficulty, compelling the agent to acquire preferences through active dialogue, perceive the environment via autonomous exploration, and perform multi-objective optimization within a complex constraint network. We evaluate state-of-the-art LLMs on S$^{3}$IT and found that they still struggle with this problem, showing an obvious gap compared with the human baseline. Results imply that LLMs have deficiencies in spatial intelligence, yet simultaneously demonstrate their ability to achieve near human-level competence in resolving conflicts that possess explicit textual cues.