Sensorimotor Attention and Language-based Regressions in Shared Latent Variables for Integrating Robot Motion Learning and LLM

TL;DR

This paper introduces a novel integration method combining robot-motion learning models with large language models using shared latent variables, enabling adaptive robot motion generation through feedback from sensorimotor and language cues.

Contribution

It proposes a new approach that updates shared latent parameters based on prediction errors, improving robot motion adaptation and generalization in tasks involving language instructions.

Findings

Enhanced position generalization in robot tasks

Improved language instruction adaptation

Effective error-driven model updates

Abstract

In recent years, studies have been actively conducted on combining large language models (LLM) and robotics; however, most have not considered end-to-end feedback in the robot-motion generation phase. The prediction of deep neural networks must contain errors, it is required to update the trained model to correspond to the real environment to generate robot motion adaptively. This study proposes an integration method that connects the robot-motion learning model and LLM using shared latent variables. When generating robot motion, the proposed method updates shared parameters based on prediction errors from both sensorimotor attention points and task language instructions given to the robot. This allows the model to search for latent parameters appropriate for the robot task efficiently. Through simulator experiments on multiple robot tasks, we demonstrated the effectiveness of our proposed method from two perspectives: position generalization and language instruction generalization abilities.