FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities

TL;DR

FUDOKI introduces a discrete flow-based unified multimodal model that surpasses autoregressive limitations, enabling iterative refinement and bidirectional context integration for visual understanding and image generation.

Contribution

It presents the first discrete flow matching approach for multimodal models, replacing autoregressive architectures with a more flexible, iterative, and self-correcting framework.

Findings

Achieves performance comparable to state-of-the-art AR-based models.

Enables iterative refinement and bidirectional context during generation.

Test-time scaling improves performance significantly.

Abstract

The rapid progress of large language models (LLMs) has catalyzed the emergence of multimodal large language models (MLLMs) that unify visual understanding and image generation within a single framework. However, most existing MLLMs rely on autoregressive (AR) architectures, which impose inherent limitations on future development, such as the raster-scan order in image generation and restricted reasoning abilities in causal context modeling. In this work, we challenge the dominance of AR-based approaches by introducing FUDOKI, a unified multimodal model purely based on discrete flow matching, as an alternative to conventional AR paradigms. By leveraging metric-induced probability paths with kinetic optimal velocities, our framework goes beyond the previous masking-based corruption process, enabling iterative refinement with self-correction capability and richer bidirectional context integration during generation. To mitigate the high cost of training from scratch, we initialize FUDOKI from pre-trained AR-based MLLMs and adaptively transition to the discrete flow matching paradigm. Experimental results show that FUDOKI achieves performance comparable to state-of-the-art AR-based MLLMs across both visual understanding and image generation tasks, highlighting its potential as a foundation for next-generation unified multimodal models. Furthermore, we show that applying test-time scaling techniques to FUDOKI yields significant performance gains, further underscoring its promise for future enhancement through reinforcement learning.