More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models

TL;DR

MERGE is a unified model that leverages pre-trained text-to-image diffusion models for both high-quality image generation and accurate depth estimation without degrading the original image generation capabilities.

Contribution

The paper introduces MERGE, a novel framework that enables seamless switching between image generation and depth estimation using a fixed pre-trained diffusion model.

Findings

MERGE achieves state-of-the-art results on multiple depth estimation benchmarks.

The model preserves the original image generation ability of the pre-trained diffusion model.

The play-and-plug framework simplifies switching between tasks with pluggable converters.

Abstract

Generative depth estimation methods leverage the rich visual priors stored in pre-trained text-to-image diffusion models, demonstrating astonishing zero-shot capability. However, parameter updates during training lead to catastrophic degradation in the image generation capability of the pre-trained model. We introduce MERGE, a unified model for image generation and depth estimation, starting from a fixed pre-trained text-to-image model. MERGE demonstrates that the pre-trained text-to-image model can do more than image generation, but also expand to depth estimation effortlessly. Specifically, MERGE introduces a play-and-plug framework that enables seamless switching between image generation and depth estimation modes through simple and pluggable converters. Meanwhile, we propose a Group Reuse Mechanism to encourage parameter reuse and improve the utilization of the additional learnable parameters. MERGE unleashes the powerful depth estimation capability of the pre-trained text-to-image model while preserving its original image generation ability. Compared to other unified models for image generation and depth estimation, MERGE achieves state-of-the-art performance across multiple depth estimation benchmarks. The code will be made available at https://github.com/H-EmbodVis/MERGE