RoboNeuron: A Middle-Layer Infrastructure for Agent-Driven Orchestration in Embodied AI

TL;DR

RoboNeuron is a middleware layer that seamlessly connects LLM agents with robot middleware like ROS2, enabling modular, reusable, and adaptable control of physical robots in embodied AI applications.

Contribution

It introduces RoboNeuron, a middleware that derives agent-callable tools from ROS schemas, supporting flexible backend transitions and modular orchestration for embodied AI.

Findings

Enables modular system orchestration with a unified interface.

Supports backend transitions without system rewiring.

Demonstrates effectiveness in simulation and hardware tasks.

Abstract

Vision-language-action (VLA) models and LLM agents have advanced rapidly, yet reliable deployment on physical robots is often hindered by an interface mismatch between agent tool APIs and robot middleware. Current implementations typically rely on ad-hoc wrappers that are difficult to reuse, and changes to the VLA backend or serving stack often necessitate extensive re-integration. We introduce RoboNeuron, a middleware layer that connects the Model Context Protocol (MCP) for LLM agents with robot middleware such as ROS2. RoboNeuron bridges these ecosystems by deriving agent-callable tools directly from ROS schemas, providing a unified execution abstraction that supports both direct commands and modular composition, and localizing backend, runtime, and acceleration-preset changes within a stable inference boundary. We evaluate RoboNeuron in simulation and on hardware through multi-platform base control, arm motion, and VLA-based grasping tasks, demonstrating that it enables modular system orchestration under a unified interface while supporting backend transitions without system rewiring. The full code implementation of this work is available at github repo: https://github.com/guanweifan/RoboNeuron