ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding

TL;DR

ROMA is a real-time omni-multimodal assistant that effectively processes continuous audio, video, and text streams for proactive and reactive interactions, advancing multimodal understanding and responsiveness.

Contribution

ROMA introduces a unified framework for real-time multimodal streaming understanding with synchronized processing, a lightweight decision head, and a comprehensive benchmark suite.

Findings

Achieves state-of-the-art in proactive tasks

Competitive performance in reactive tasks

Validated robustness across 12 benchmarks

Abstract

Recent Omni-multimodal Large Language Models show promise in unified audio, vision, and text modeling. However, streaming audio-video understanding remains challenging, as existing approaches suffer from disjointed capabilities: they typically exhibit incomplete modality support or lack autonomous proactive monitoring. To address this, we present ROMA, a real-time omni-multimodal assistant for unified reactive and proactive interaction. ROMA processes continuous inputs as synchronized multimodal units, aligning dense audio with discrete video frames to handle granularity mismatches. For online decision-making, we introduce a lightweight speak head that decouples response initiation from generation to ensure precise triggering without task conflict. We train ROMA with a curated streaming dataset and a two-stage curriculum that progressively optimizes for streaming format adaptation and proactive responsiveness. To standardize the fragmented evaluation landscape, we reorganize diverse benchmarks into a unified suite covering both proactive (alert, narration) and reactive (QA) settings. Extensive experiments across 12 benchmarks demonstrate ROMA achieves state-of-the-art performance on proactive tasks while competitive in reactive settings, validating its robustness in unified real-time omni-multimodal understanding.