A User Interface for Sense-making of the Reasoning Process while Interacting with Robots

TL;DR

This paper presents a user interface that visualizes robot reasoning processes by linking natural language commands, knowledge graph queries, and real-world 3D mapping to improve understanding and supervision of robotic actions.

Contribution

The authors developed an interactive interface that connects natural language commands with knowledge graph reasoning and real-world visualization, enhancing expert understanding of robot decision-making.

Findings

User feedback indicates improved understanding of robot reasoning.

System facilitates testing and modification of robot knowledge and actions.

Positive reception from robotic experts after 3 weeks of use.

Abstract

As knowledge graph has the potential to bridge the gap between commonsense knowledge and reasoning over actionable capabilities of mobile robotic platforms, incorporating knowledge graph into robotic system attracted increasing attention in recent years. Previously, graph visualization has been used wildly by developers to make sense of knowledge representations. However, due to lacking the link between abstract knowledge of the real-world environment and the robot's actions, transitional visualization tools are incompatible for expert-user to understand, test, supervise and modify the graph-based reasoning system with the embodiment of the robots. Therefore, we developed an interface which enables robotic experts to send commands to the robot in natural language, then interface visualizes the procedures of the robot mapping the command to the functions for querying in the commonsense knowledge database, links the result to the real world instances in a 3D map and demonstrate the execution of the robot from the first-person perspective of the robot. After 3 weeks of usage of the system by robotic experts in their daily development, some feedback was collected, which provides insight for designing such systems.