SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness

TL;DR

SpatialFusion introduces a framework that integrates 3D geometric understanding into unified image generation models, enhancing spatial awareness and coherence in generated images.

Contribution

It employs a Mixture-of-Transformers architecture with a spatial transformer and depth adapter to incorporate explicit geometric guidance into diffusion-based image generation.

Findings

Outperforms leading models like GPT-4o on spatial benchmarks.

Enhances performance in text-to-image and image editing tasks.

Maintains negligible inference overhead.

Abstract

Recent unified image generation models have achieved remarkable success by employing MLLMs for semantic understanding and diffusion backbones for image generation. However, these models remain fundamentally limited in spatially-aware tasks due to a lack of intrinsic spatial understanding and the absence of explicit geometric guidance during generation. In this paper, we propose SpatialFusion, a novel framework that internalizes 3D geometric awareness into unified image generation models. Specifically, we first employ a Mixture-of-Transformers (MoT) architecture to augment the MLLM with a parallel spatial transformer to enhance 3D geometric modeling capability. By sharing self-attention with the MLLM, the spatial transformer learns to derive metric-depth maps of target images from rich semantic contexts. These explicit geometric scaffolds are then injected into the diffusion backbone through a specialized depth adapter, providing precise spatial constraints for spatially-coherent image generation. Through a progressive two-stage training strategy, SpatialFusion significantly enhances performance on spatially-aware benchmarks, notably outperforming leading models such as GPT-4o. Additionally, it achieves generalized performance gains across both text-to-image generation and image editing scenarios, all while maintaining negligible inference overhead.