InstructRobot: A Model-Free Framework for Mapping Natural Language Instructions into Robot Motion

TL;DR

InstructRobot is a novel, model-free framework that translates natural language instructions into robot motions using reinforcement learning, capable of handling complex robots without large datasets or detailed kinematic models.

Contribution

It introduces a reinforcement learning-based approach that jointly learns language representations and inverse kinematics, enabling flexible, dataset-free robot instruction mapping.

Findings

Successfully applied to a 26-joint robot in manipulation tasks

Demonstrates robustness and adaptability in realistic environments

Operates without large datasets or prior kinematic knowledge

Abstract

The ability to communicate with robots using natural language is a significant step forward in human-robot interaction. However, accurately translating verbal commands into physical actions is promising, but still presents challenges. Current approaches require large datasets to train the models and are limited to robots with a maximum of 6 degrees of freedom. To address these issues, we propose a framework called InstructRobot that maps natural language instructions into robot motion without requiring the construction of large datasets or prior knowledge of the robot's kinematics model. InstructRobot employs a reinforcement learning algorithm that enables joint learning of language representations and inverse kinematics model, simplifying the entire learning process. The proposed framework is validated using a complex robot with 26 revolute joints in object manipulation tasks, demonstrating its robustness and adaptability in realistic environments. The framework can be applied to any task or domain where datasets are scarce and difficult to create, making it an intuitive and accessible solution to the challenges of training robots using linguistic communication. Open source code for the InstructRobot framework and experiments can be accessed at https://github.com/icleveston/InstructRobot.