Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis

TL;DR

Marigold is a cost-effective method that adapts large pretrained diffusion models for dense image analysis tasks, achieving state-of-the-art zero-shot performance with minimal training.

Contribution

It introduces a novel fine-tuning protocol that leverages pretrained latent diffusion models for dense image analysis, requiring minimal modifications and synthetic data.

Findings

Achieves state-of-the-art zero-shot results on multiple dense prediction tasks.

Requires only small synthetic datasets and minimal training time.

Demonstrates effective adaptation of diffusion models for image analysis.

Abstract

The success of deep learning in computer vision over the past decade has hinged on large labeled datasets and strong pretrained models. In data-scarce settings, the quality of these pretrained models becomes crucial for effective transfer learning. Image classification and self-supervised learning have traditionally been the primary methods for pretraining CNNs and transformer-based architectures. Recently, the rise of text-to-image generative models, particularly those using denoising diffusion in a latent space, has introduced a new class of foundational models trained on massive, captioned image datasets. These models' ability to generate realistic images of unseen content suggests they possess a deep understanding of the visual world. In this work, we present Marigold, a family of conditional generative models and a fine-tuning protocol that extracts the knowledge from pretrained latent diffusion models like Stable Diffusion and adapts them for dense image analysis tasks, including monocular depth estimation, surface normals prediction, and intrinsic decomposition. Marigold requires minimal modification of the pre-trained latent diffusion model's architecture, trains with small synthetic datasets on a single GPU over a few days, and demonstrates state-of-the-art zero-shot generalization. Project page: https://marigoldcomputervision.github.io