OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows

TL;DR

OmniFlow is a versatile multi-modal generative model that extends rectified flow techniques to handle any-to-any tasks across text, image, and audio modalities, with flexible control and efficient training.

Contribution

It extends rectified flow to multi-modal settings, introduces a guidance mechanism for modality alignment, and proposes a scalable architecture for joint text, image, and audio generation.

Findings

Outperforms previous models on text-to-image and text-to-audio tasks.

Provides a flexible guidance mechanism for modality control.

Offers insights into optimizing multi-modal rectified flow transformers.

Abstract

We introduce OmniFlow, a novel generative model designed for any-to-any generation tasks such as text-to-image, text-to-audio, and audio-to-image synthesis. OmniFlow advances the rectified flow (RF) framework used in text-to-image models to handle the joint distribution of multiple modalities. It outperforms previous any-to-any models on a wide range of tasks, such as text-to-image and text-to-audio synthesis. Our work offers three key contributions: First, we extend RF to a multi-modal setting and introduce a novel guidance mechanism, enabling users to flexibly control the alignment between different modalities in the generated outputs. Second, we propose a novel architecture that extends the text-to-image MMDiT architecture of Stable Diffusion 3 and enables audio and text generation. The extended modules can be efficiently pretrained individually and merged with the vanilla text-to-image MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design choices of rectified flow transformers for large-scale audio and text generation, providing valuable insights into optimizing performance across diverse modalities. The Code will be available at https://github.com/jacklishufan/OmniFlows.