CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision

TL;DR

This paper introduces CLIP-RT, a vision-language model that learns robotic manipulation policies from natural language supervision, enabling non-experts to teach robots skills efficiently and outperforming existing models in success rates.

Contribution

We propose a novel framework for collecting natural language-supervised robotic data and introduce CLIP-RT, a model that learns language-conditioned visuomotor policies from this data.

Findings

CLIP-RT outperforms OpenVLA by 24% in success rates.

CLIP-RT achieves 93.1% success on LIBERO benchmark.

CLIP-RT operates at 163 Hz inference throughput.

Abstract

Teaching robots desired skills in real-world environments remains challenging, especially for non-experts. A key bottleneck is that collecting robotic data often requires expertise or specialized hardware, limiting accessibility and scalability. We posit that natural language offers an intuitive and accessible interface for robot learning. To this end, we study two aspects: (1) enabling non-experts to collect robotic data through natural language supervision (e.g., "move the arm to the right") and (2) training robot policies directly from this supervision. Specifically, we introduce a data collection framework that collects robot demonstrations based on natural language supervision and further augments these demonstrations. We then present CLIP-RT, a new vision-language-action (VLA) model that learns language-conditioned visuomotor policies from this supervision. CLIP-RT adapts the pretrained CLIP model and learns to predict language-based motion primitives via contrastive imitation learning. We train CLIP-RT on the Open X-Embodiment dataset and finetune it on in-domain data collected by our framework. In real-world evaluations, CLIP-RT demonstrates strong capabilities in learning novel manipulation skills, outperforming OpenVLA (7B parameters) by 24% in average success rates, while using 7x fewer parameters (1B). We further assess CLIP-RT's capabilities in few-shot generalization and collaborative scenarios involving large pretrained models or humans. In simulated environments, CLIP-RT also yields strong performance, achieving a 93.1% average success rate on the LIBERO benchmark with an inference throughput of 163 Hz.