Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models

TL;DR

This paper introduces a novel parallel streaming framework for multimodal large language models, enabling real-time video understanding by relaxing positional encoding constraints, which significantly reduces latency and improves efficiency.

Contribution

The authors propose a new parallel streaming approach with three designs that allow simultaneous perception and generation, overcoming the limitations of traditional sequential methods in real-time video processing.

Findings

Group-Decoupled design achieves best efficiency-performance balance.

Framework yields up to 2x acceleration in real-time processing.

Significantly reduces latency while maintaining high accuracy.

Abstract

Multimodal Large Language Models (MLLMs) have achieved strong performance across many tasks, yet most systems remain limited to offline inference, requiring complete inputs before generating outputs. Recent streaming methods reduce latency by interleaving perception and generation, but still enforce a sequential perception-generation cycle, limiting real-time interaction. In this work, we target a fundamental bottleneck that arises when extending MLLMs to real-time video understanding: the global positional continuity constraint imposed by standard positional encoding schemes. While natural in offline inference, this constraint tightly couples perception and generation, preventing effective input-output parallelism. To address this limitation, we propose a parallel streaming framework that relaxes positional continuity through three designs: Overlapped, Group-Decoupled, and Gap-Isolated. These designs enable simultaneous perception and generation, allowing the model to process incoming inputs while producing responses in real time. Extensive experiments reveal that Group-Decoupled achieves the best efficiency-performance balance, maintaining high fluency and accuracy while significantly reducing latency. We further show that the proposed framework yields up to 2x acceleration under balanced perception-generation workloads, establishing a principled pathway toward speak-while-watching real-time systems. We make all our code publicly available: https://github.com/EIT-NLP/Speak-While-Watching.