RACAS: Controlling Diverse Robots With a Single Agentic System

TL;DR

RACAS introduces a robot-agnostic, language-based control system using large language and vision models, enabling diverse robots to perform tasks without platform-specific modifications, simplifying robotic prototyping.

Contribution

The paper presents RACAS, a novel agentic architecture that controls various robots through natural language modules, eliminating the need for retraining or code modifications across platforms.

Findings

Successfully controlled diverse robots with RACAS

Consistently solved tasks across different robotic platforms

Demonstrated reduction in prototyping barriers for robotics

Abstract

Many robotic platforms expose an API through which external software can command their actuators and read their sensors. However, transitioning from these low-level interfaces to high-level autonomous behaviour requires a complicated pipeline, whose components demand distinct areas of expertise. Existing approaches to bridging this gap either require retraining for every new embodiment or have only been validated across structurally similar platforms. We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a Memory Curator) communicate exclusively through natural language to provide closed-loop robot control. RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms. We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential of agentic AI to substantially reduce the barrier to prototyping robotic solutions.