Bringing the RT-1-X Foundation Model to a SCARA robot

TL;DR

This paper investigates the ability of the RT-1-X foundation model to generalize to a new SCARA robot, finding that zero-shot transfer fails but fine-tuning enables skill transfer for specific tasks.

Contribution

It demonstrates that while RT-1-X does not zero-shot generalize to new robot types, fine-tuning by demonstration can transfer learned skills to unseen robots.

Findings

RT-1-X does not zero-shot generalize to SCARA robots.

Fine-tuning enables transfer of pickup skills.

Object-specific knowledge is not transferred through fine-tuning.

Abstract

Traditional robotic systems require specific training data for each task, environment, and robot form. While recent advancements in machine learning have enabled models to generalize across new tasks and environments, the challenge of adapting these models to entirely new settings remains largely unexplored. This study addresses this by investigating the generalization capabilities of the RT-1-X robotic foundation model to a type of robot unseen during its training: a SCARA robot from UMI-RTX. Initial experiments reveal that RT-1-X does not generalize zero-shot to the unseen type of robot. However, fine-tuning of the RT-1-X model by demonstration allows the robot to learn a pickup task which was part of the foundation model (but learned for another type of robot). When the robot is presented with an object that is included in the foundation model but not in the fine-tuning dataset, it demonstrates that only the skill, but not the object-specific knowledge, has been transferred.