RAI: Flexible Agent Framework for Embodied AI

TL;DR

RAI is a versatile framework that facilitates the development of embodied multi-agent systems in robotics, integrating language models, robotic stacks, and simulations for rapid prototyping and evaluation.

Contribution

It introduces a flexible framework for embodied AI that seamlessly integrates agents with robotic systems, simulations, and language models, enabling multi-agent system development and testing.

Findings

Successful deployment on physical robot Husarion ROSBot XL

Effective in robot arm and tractor simulation tasks

Identified limitations of generative models in embodied AI

Abstract

With an increase in the capabilities of generative language models, a growing interest in embodied AI has followed. This contribution introduces RAI - a framework for creating embodied Multi Agent Systems for robotics. The proposed framework implements tools for Agents' integration with robotic stacks, Large Language Models, and simulations. It provides out-of-the-box integration with state-of-the-art systems like ROS 2. It also comes with dedicated mechanisms for the embodiment of Agents. These mechanisms have been tested on a physical robot, Husarion ROSBot XL, which was coupled with its digital twin, for rapid prototyping. Furthermore, these mechanisms have been deployed in two simulations: (1) robot arm manipulator and (2) tractor controller. All of these deployments have been evaluated in terms of their control capabilities, effectiveness of embodiment, and perception ability. The proposed framework has been used successfully to build systems with multiple agents. It has demonstrated effectiveness in all the aforementioned tasks. It also enabled identifying and addressing the shortcomings of the generative models used for embodied AI.