How Does a Virtual Agent Decide Where to Look? Symbolic Cognitive Reasoning for Embodied Head Rotation

TL;DR

This paper introduces SCORE, a symbolic reasoning framework enabling virtual agents to perform context-aware, human-like head movements by integrating scene perception and motivation-driven planning without task-specific training.

Contribution

SCORE is a novel, data-agnostic framework that combines symbolic reasoning, vision-language perception, and language models to produce realistic, motivation-driven head rotations in virtual agents.

Findings

SCORE generalizes well to unseen scenes and multi-agent environments.

The framework captures five key human head movement drivers.

Agents exhibit more realistic and context-aware gaze behaviors.

Abstract

Natural head rotation is critical for believable embodied virtual agents, yet this micro-level behavior remains largely underexplored. While head-rotation prediction algorithms could, in principle, reproduce this behavior, they typically focus on visually salient stimuli and overlook the cognitive motives that guide head rotation. This yields agents that look at conspicuous objects while overlooking obstacles or task-relevant cues, diminishing realism in a virtual environment. We introduce SCORE, a Symbolic Cognitive Reasoning framework for Embodied Head Rotation, a data-agnostic framework that produces context-aware head movements without task-specific training or hand-tuned heuristics. A controlled VR study (N=20) identifies five motivational drivers of human head movements: Interest, Information Seeking, Safety, Social Schema, and Habit. SCORE encodes these drivers as symbolic predicates, perceives the scene with a Vision-Language Model (VLM), and plans head poses with a Large Language Model (LLM). The framework employs a hybrid workflow: the VLM-LLM reasoning is executed offline, after which a lightweight FastVLM performs online validation to suppress hallucinations while maintaining responsiveness to scene dynamics. The result is an agent that predicts not only where to look but also why, generalizing to unseen scenes and multi-agent crowds while retaining behavioral plausibility.