ChatUMM: Robust Context Tracking for Conversational Interleaved Generation

TL;DR

ChatUMM introduces a novel conversational unified multimodal model that excels at robust context tracking and interleaved multimodal generation, enabling more natural and continuous multi-turn dialogues in visual and textual tasks.

Contribution

It proposes an interleaved multi-turn training strategy and a systematic data synthesis pipeline to enhance continuous dialogue capabilities in unified multimodal models.

Findings

Achieves state-of-the-art performance on visual understanding benchmarks.

Demonstrates superior robustness in complex multi-turn scenarios.

Maintains competitive fidelity in text-to-image generation.

Abstract

Unified multimodal models (UMMs) have achieved remarkable progress yet remain constrained by a single-turn interaction paradigm, effectively functioning as solvers for independent requests rather than assistants in continuous dialogue. To bridge this gap, we present ChatUMM. As a conversational unified model, it excels at robust context tracking to sustain interleaved multimodal generation. ChatUMM derives its capabilities from two key innovations: an interleaved multi-turn training strategy that models serialized text-image streams as a continuous conversational flow, and a systematic conversational data synthesis pipeline. This pipeline transforms a diverse set of standard single-turn datasets into fluid dialogues through three progressive stages: constructing basic stateful dialogues, enforcing long-range dependency resolution via ``distractor'' turns with history-dependent query rewriting, and synthesizing naturally interleaved multimodal responses. Extensive evaluations demonstrate that ChatUMM achieves state-of-the-art performance among open-source unified models on visual understanding and instruction-guided editing benchmarks, while maintaining competitive fidelity in text-to-image generation. Notably, ChatUMM exhibits superior robustness in complex multi-turn scenarios, ensuring fluid, context-aware dialogues.