XARP Tools: An Extended Reality Platform for Humans and AI Agents

TL;DR

XARP is an open-source Python toolkit that simplifies XR-AI prototyping by enabling seamless communication between Python AI logic and Unity-based XR applications, improving development speed and reducing complexity.

Contribution

The paper introduces XARP, a novel Python-based toolkit that bridges XR and AI development ecosystems, facilitating rapid prototyping and reducing engineering friction.

Findings

Faster iteration and easier setup compared to traditional workflows.

AI agents using XARP consume 19% fewer tokens than C# Unity code.

Tracking data streaming achieved near device refresh rate of 72 FPS.

Abstract

Building XR-AI research prototypes requires navigating two largely separate ecosystems. Mainstream XR development relies on C#/C++ and game engines, while AI development is centered on Python. This toolchain fragmentation slows down contributions to human-AI spatial interaction research. To broaden access to XR development in the Python ecosystem, we present XARP (XR Agent-ready Remote Procedures), a toolkit for rapid XR-AI prototyping in Python. XARP application logic runs on a Python server and controls a Unity client through WebSocket messages. This architecture enables compatibility with multiple client platforms and live reloading of application code without client redeployment. XARP is available to humans as a library and to AI agents as callable tools and through Model Context Protocol. We designed XARP through formative case studies and refined it through an early acceptance evaluation with 24 XR and AI developers and a six-week longitudinal study with two developers building an independent research project. Potential users expected the toolkit to improve their performance and facilitate development. Sustained use confirmed faster iteration and easier setup compared to conventional XR workflows, with asset-intensive and performance-critical projects emerging as the clearest limitations. Technical benchmarks show that hand and head tracking data streaming was close to the device refresh rate of 72 FPS, and that AI agents using XARP consumed 19% fewer tokens than those writing equivalent C# Unity code. Beyond broadening access to XR development, XARP reduces engineering friction in spatial computing research and opens new pathways for AI agents to participate in XR application development. XARP is open source and available at https://github.com/hal-ucsb/xarp.