UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation

TL;DR

UniModel introduces a unified visual-only diffusion framework that jointly supports multimodal understanding and generation by translating all modalities into a shared pixel space, enabling versatile vision-language tasks.

Contribution

The paper presents a novel pixel-to-pixel diffusion model that unifies multimodal tasks by representing text and images in a shared visual space, eliminating modality discrepancies.

Findings

Strong cross-modal alignment demonstrated in experiments.

Emergent controllability such as cycle-consistent image-caption-image loops.

Effective for both text-to-image synthesis and image-to-text understanding.

Abstract

We present UniModel, a unified generative model that jointly supports visual understanding and visual generation within a single pixel-to-pixel diffusion framework. Our goal is to achieve unification along three axes: the model, the tasks, and the representations. At the representation level, we eliminate modality discrepancies by mapping both text and images into a shared visual space: textual prompts are rendered as painted text images on a clean canvas, and all inputs and outputs are treated purely as RGB pixels. This yields a fully vision-native formulation of multimodal learning. At the task level, a broad range of vision-language problems are cast as pixel-to-pixel transformations in this visual space. For understanding tasks, the model takes an RGB image and produces a painted text image that visually encodes the semantic prediction. For generation tasks, painted text images serve as visual conditions that guide realistic and semantically aligned image synthesis. Captioning and text-to-image generation thus become different directions of the same underlying visual translation process. At the model level, we instantiate a single Unified Diffusion Transformer trained with rectified flow in pixel space. A shared backbone jointly learns bidirectional mappings between natural images and painted text images, with lightweight task embeddings to specify the desired direction. Experiments on text-to-image synthesis and image-to-text understanding demonstrate strong cross-modal alignment and emergent controllability such as cycle-consistent image-caption-image loops. Our initial exploration suggests that unifying model, tasks, and representations in a single visual space is a promising paradigm for general-purpose multimodal intelligence.