Remote Life Support Robot Interface System for Global Task Planning and Local Action Expansion Using Foundation Models

TL;DR

This paper introduces a robot interface system that uses foundation models to handle uncertain instructions through template variables, enabling effective global planning and local action expansion in real-life support tasks.

Contribution

It presents a novel framework integrating template variables and prompt generation for improved communication and task execution in language-guided robot systems.

Findings

Effective communication of uncertain info via template variables

Successful application to real-life support robot tasks

Enhanced task planning and local action expansion

Abstract

Robot systems capable of executing tasks based on language instructions have been actively researched. It is challenging to convey uncertain information that can only be determined on-site with a single language instruction to the robot. In this study, we propose a system that includes ambiguous parts as template variables in language instructions to communicate the information to be collected and the options to be presented to the robot for predictable uncertain events. This study implements prompt generation for each robot action function based on template variables to collect information, and a feedback system for presenting and selecting options based on template variables for user-to-robot communication. The effectiveness of the proposed system was demonstrated through its application to real-life support tasks performed by the robot.