One Diffusion to Generate Them All

TL;DR

OneDiffusion is a unified large-scale diffusion model capable of performing diverse image synthesis and understanding tasks, including conditional generation, image editing, and multi-view analysis, with a simple training approach that enhances scalability and generalization.

Contribution

It introduces a versatile diffusion framework that supports multiple tasks without specialized architectures, enabling scalable multi-task training and smooth adaptation to various resolutions.

Findings

Achieves competitive results across diverse tasks

Supports multi-view generation and camera pose estimation

Operates effectively with relatively small training datasets

Abstract

We introduce OneDiffusion, a versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. It enables conditional generation from inputs such as text, depth, pose, layout, and semantic maps, while also handling tasks like image deblurring, upscaling, and reverse processes such as depth estimation and segmentation. Additionally, OneDiffusion allows for multi-view generation, camera pose estimation, and instant personalization using sequential image inputs. Our model takes a straightforward yet effective approach by treating all tasks as frame sequences with varying noise scales during training, allowing any frame to act as a conditioning image at inference time. Our unified training framework removes the need for specialized architectures, supports scalable multi-task training, and adapts smoothly to any resolution, enhancing both generalization and scalability. Experimental results demonstrate competitive performance across tasks in both generation and prediction such as text-to-image, multiview generation, ID preservation, depth estimation and camera pose estimation despite relatively small training dataset. Our code and checkpoint are freely available at https://github.com/lehduong/OneDiffusion