XR$^3$: An Extended Reality Platform for Social-Physical Human-Robot Interaction

TL;DR

XR$^3$ is an innovative virtual reality platform enabling realistic social-physical human-robot interactions by synchronizing real-time operator movements with a virtual robot in shared physical space.

Contribution

The paper introduces XR$^3$, a novel dual-VR-headset system that allows co-located human-robot interaction studies with synchronized physical contact and expressive virtual embodiments.

Findings

Enables realistic touch-based robot interactions in VR.

Supports precise motion retargeting and nonverbal cue manipulation.

Reduces barriers to prototyping embodied robot behaviors.

Abstract

Social-physical human-robot interaction (spHRI) is difficult to study: building and programming robots that integrate multiple interaction modalities is costly and slow, while VR-based prototypes often lack physical contact, breaking users' visuo-tactile expectations. We present XR$^3$, a co-located dual-VR-headset platform for HRI research in which an attendee and a hidden operator share the same physical space while experiencing different virtual embodiments. The attendee sees an expressive virtual robot that interacts face-to-face in a shared virtual environment. In real time, the robot's upper-body motion, head and gaze behavior, and facial expressions are mapped from the operator's tracked limbs and face signals. Because the operator is co-present and calibrated in the same coordinate frame, the operator can also touch the attendee, enabling perceived robot touch synchronized with the robot's visible hands. Finger and hand motion is mapped to the robot avatar using inverse kinematics to support precise contact. Beyond motion retargeting, XR$^3$ supports social retargeting of multiple nonverbal cues that can be experimentally varied while keeping physical interaction constant. We detail the system design and calibration, and demonstrate the platform in a touch-based Wizard-of-Oz study, lowering the barrier to prototyping and evaluating embodied, contact-based robot behaviors.