CognitiveOS: Large Multimodal Model based System to Endow Any Type of Robot with Generative AI

TL;DR

CognitiveOS is a modular, transformer-based operating system for cognitive robots that enhances adaptability, scalability, and task understanding across diverse platforms, facilitating advanced real-world robotic applications.

Contribution

This work introduces the first modular operating system for cognitive robots, enabling seamless integration, configuration, and scalability of modules for diverse robotic tasks.

Findings

System outperforms CognitiveDog by 15% in reasoning tasks

Achieves 77% highest reasoning accuracy to date

Demonstrates adaptability across various robotic platforms

Abstract

This paper introduces CognitiveOS, the first operating system designed for cognitive robots capable of functioning across diverse robotic platforms. CognitiveOS is structured as a multi-agent system comprising modules built upon a transformer architecture, facilitating communication through an internal monologue format. These modules collectively empower the robot to tackle intricate real-world tasks. The paper delineates the operational principles of the system along with descriptions of its nine distinct modules. The modular design endows the system with distinctive advantages over traditional end-to-end methodologies, notably in terms of adaptability and scalability. The system's modules are configurable, modifiable, or deactivatable depending on the task requirements, while new modules can be seamlessly integrated. This system serves as a foundational resource for researchers and developers in the cognitive robotics domain, alleviating the burden of constructing a cognitive robot system from scratch. Experimental findings demonstrate the system's advanced task comprehension and adaptability across varied tasks, robotic platforms, and module configurations, underscoring its potential for real-world applications. Moreover, in the category of Reasoning it outperformed CognitiveDog (by 15%) and RT2 (by 31%), achieving the highest to date rate of 77%. We provide a code repository and dataset for the replication of CognitiveOS: link will be provided in camera-ready submission.