From Woofs to Words: Towards Intelligent Robotic Guide Dogs with Verbal Communication

TL;DR

This paper presents a novel dialog system for robotic guide dogs that uses large language models to verbalize navigation plans and scenes, enhancing communication and collaboration with visually impaired users.

Contribution

It introduces a new approach to grounding language in dynamic environments and improving spatial awareness in assistive robotic guide dogs using LLMs.

Findings

Verbalization strategies impact user understanding and trust.

The system improves navigation efficiency and accuracy.

Human studies validate the effectiveness of verbal communication.

Abstract

Assistive robotics is an important subarea of robotics that focuses on the well-being of people with disabilities. A robotic guide dog is an assistive quadruped robot that helps visually impaired people in obstacle avoidance and navigation. Enabling language capabilities for robotic guide dogs goes beyond naively adding an existing dialog system onto a mobile robot. The novel challenges include grounding language in the dynamically changing environment and improving spatial awareness for the human handler. To address those challenges, we develop a novel dialog system for robotic guide dogs that uses LLMs to verbalize both navigational plans and scenes. The goal is to enable verbal communication for collaborative decision-making within the handler-robot team. In experiments, we conducted a human study to evaluate different verbalization strategies and a simulation study to assess the efficiency and accuracy in navigation tasks.