RoboEgo System Card: An Omnimodal Model with Native Full Duplexity

TL;DR

RoboEgo is a unified omnimodal model that supports native full duplexity, enabling rapid, natural multimodal interactions with low latency, advancing AI's ability to process real-world, multimodal conversations.

Contribution

It introduces RoboEgo, a novel model architecture that natively supports full duplex multimodal processing with low latency, addressing key challenges in real-time, multimodal AI interactions.

Findings

Achieves a duplex latency of 80 ms.

Demonstrates superior responsiveness and speech naturalness.

Maintains content quality comparable to semi-duplex models.

Abstract

Humans naturally process real-world multimodal information in a full-duplex manner. In artificial intelligence, replicating this capability is essential for advancing model development and deployment, particularly in embodied contexts. The development of multimodal models faces two primary challenges: (1) effectively handling more than three modalities-such as vision, audio, and text; and (2) delivering full-duplex responses to rapidly evolving human instructions. To facilitate research on models that support both omnimodal processing and full duplexity, we present RoboEgo (alias: FLM-Ego), a unified model system designed to address both challenges. RoboEgo incorporates a backbone architecture and algorithms that natively support full duplexity, achieving a theoretical duplex latency of 80 ms. In streaming visually grounded conversations under real-world conditions, RoboEgo exhibits superior responsiveness and speech naturalness, while maintaining comparable content qualities to state-of-the-art semi-duplex omnimodal models-a feat previously considered unattainable by native full-duplex systems.