PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision

TL;DR

This paper introduces PerspAct, a method that combines perspective-taking and active vision to improve large language models' ability to collaborate effectively in multi-agent scenarios, especially in complex environments requiring viewpoint reasoning.

Contribution

The study extends the Director task with active visual exploration and demonstrates that explicit perspective cues and active strategies enhance LLM interpretative accuracy and collaboration.

Findings

Active exploration improves interpretative accuracy.

Explicit perspective cues enhance collaboration.

ReAct-style reasoning benefits multi-agent understanding.

Abstract

Recent advances in Large Language Models (LLMs) and multimodal foundation models have significantly broadened their application in robotics and collaborative systems. However, effective multi-agent interaction necessitates robust perspective-taking capabilities, enabling models to interpret both physical and epistemic viewpoints. Current training paradigms often neglect these interactive contexts, resulting in challenges when models must reason about the subjectivity of individual perspectives or navigate environments with multiple observers. This study evaluates whether explicitly incorporating diverse points of view using the ReAct framework, an approach that integrates reasoning and acting, can enhance an LLM's ability to understand and ground the demands of other agents. We extend the classic Director task by introducing active visual exploration across a suite of seven scenarios of increasing perspective-taking complexity. These scenarios are designed to challenge the agent's capacity to resolve referential ambiguity based on visual access and interaction, under varying state representations and prompting strategies, including ReAct-style reasoning. Our results demonstrate that explicit perspective cues, combined with active exploration strategies, significantly improve the model's interpretative accuracy and collaborative effectiveness. These findings highlight the potential of integrating active perception with perspective-taking mechanisms in advancing LLMs' application in robotics and multi-agent systems, setting a foundation for future research into adaptive and context-aware AI systems.