Explainable OOHRI: Communicating Robot Capabilities and Limitations as Augmented Reality Affordances

TL;DR

This paper introduces X-OOHRI, an augmented reality interface that visually communicates robot capabilities and limitations to users, enhancing human-robot interaction and understanding.

Contribution

It presents a novel AR-based system that encodes robot and object properties for real-time explanation and manipulation, improving transparency in human-robot interaction.

Findings

Participants issued object-oriented commands effectively.

Users developed accurate mental models of robot limitations.

The system facilitated mixed-initiative resolution.

Abstract

Human interaction is essential for issuing personalized instructions and assisting robots when failure is likely. However, robots remain largely black boxes, offering users little insight into their evolving capabilities and limitations. To address this gap, we present explainable object-oriented HRI (X-OOHRI), an augmented reality (AR) interface that conveys robot action possibilities and constraints through visual signifiers, radial menus, color coding, and explanation tags. Our system encodes object properties and robot limits into object-oriented structures using a vision-language model, allowing explanation generation on the fly and direct manipulation of virtual twins spatially aligned within a simulated environment. We integrate the end-to-end pipeline with a physical robot and showcase diverse use cases ranging from low-level pick-and-place to high-level instructions. Finally, we evaluate X-OOHRI through a user study and find that participants effectively issue object-oriented commands, develop accurate mental models of robot limitations, and engage in mixed-initiative resolution.