What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?

TL;DR

This paper investigates key factors affecting the transfer efficiency of diffusion models for dense perception tasks, emphasizing data quality, the stochastic nature of diffusion, and supervision strategies, leading to a fast, effective fine-tuning paradigm called GenPercept.

Contribution

It provides a comprehensive analysis of diffusion model fine-tuning for perception tasks and introduces GenPercept, a one-step deterministic fine-tuning method with improved speed and performance.

Findings

High-quality fine-tuning data is crucial for perception tasks.

Stochastic diffusion models slightly hinder deterministic perception performance.

Task-specific image-level supervision enhances fine-grained details.

Abstract

Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I diffusion models for dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step stochastic diffusion mechanism, training strategy, inference ensemble strategy, and fine-tuning data quality. In this work, we conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) The stochastic nature of diffusion models has a slightly negative impact on deterministic visual perception tasks. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task-specific image-level supervision is beneficial to enhance fine-grained details. These observations culminate in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailed for dense visual perception tasks. Different from the previous multi-step methods, our paradigm has a much faster inference speed, and can be seamlessly integrated with customized perception decoders and loss functions for image-level supervision, which is critical to improving the fine-grained details of predictions. Comprehensive experiments on diverse dense visual perceptual tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting, are performed to demonstrate the remarkable adaptability and effectiveness of our proposed method.