Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction

TL;DR

Lotus is a diffusion-based model tailored for dense prediction tasks that predicts annotations directly, simplifies the diffusion process, and achieves state-of-the-art zero-shot performance with high efficiency.

Contribution

The paper introduces Lotus, a novel diffusion-based dense prediction model that predicts annotations directly and reformulates the diffusion process into a single step for improved speed and accuracy.

Findings

Achieves state-of-the-art zero-shot depth and normal estimation.

Significantly faster inference compared to existing diffusion methods.

Improves practical applications like 3D reconstruction.

Abstract

Leveraging the visual priors of pre-trained text-to-image diffusion models offers a promising solution to enhance zero-shot generalization in dense prediction tasks. However, existing methods often uncritically use the original diffusion formulation, which may not be optimal due to the fundamental differences between dense prediction and image generation. In this paper, we provide a systemic analysis of the diffusion formulation for the dense prediction, focusing on both quality and efficiency. And we find that the original parameterization type for image generation, which learns to predict noise, is harmful for dense prediction; the multi-step noising/denoising diffusion process is also unnecessary and challenging to optimize. Based on these insights, we introduce Lotus, a diffusion-based visual foundation model with a simple yet effective adaptation protocol for dense prediction. Specifically, Lotus is trained to directly predict annotations instead of noise, thereby avoiding harmful variance. We also reformulate the diffusion process into a single-step procedure, simplifying optimization and significantly boosting inference speed. Additionally, we introduce a novel tuning strategy called detail preserver, which achieves more accurate and fine-grained predictions. Without scaling up the training data or model capacity, Lotus achieves SoTA performance in zero-shot depth and normal estimation across various datasets. It also enhances efficiency, being significantly faster than most existing diffusion-based methods. Lotus' superior quality and efficiency also enable a wide range of practical applications, such as joint estimation, single/multi-view 3D reconstruction, etc. Project page: https://lotus3d.github.io/.