DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks

TL;DR

DICEPTION is a versatile diffusion-based visual perception model that efficiently handles multiple tasks with minimal training data and computational resources, achieving near state-of-the-art performance.

Contribution

The paper introduces DICEPTION, a generalist diffusion model that re-purposes pre-trained text-to-image diffusion models for diverse perception tasks with low data and computational costs.

Findings

Achieves performance comparable to SOTA models with only 0.06% of their data

Requires fine-tuning on as few as 50 images for new tasks

Subtle classifier-free guidance improves depth and normal estimation

Abstract

This paper's primary objective is to develop a robust generalist perception model capable of addressing multiple tasks under constraints of computational resources and limited training data. We leverage text-to-image diffusion models pre-trained on billions of images and successfully introduce our DICEPTION, a visual generalist model. Exhaustive evaluations demonstrate that DICEPTION effectively tackles diverse perception tasks, even achieving performance comparable to SOTA single-task specialist models. Specifically, we achieve results on par with SAM-vit-h using only 0.06% of their data (e.g., 600K vs.\ 1B pixel-level annotated images). We designed comprehensive experiments on architectures and input paradigms, demonstrating that the key to successfully re-purposing a single diffusion model for multiple perception tasks lies in maximizing the preservation of the pre-trained model's prior knowledge. Consequently, DICEPTION can be trained with substantially lower computational costs than conventional models requiring training from scratch. Furthermore, adapting DICEPTION to novel tasks is highly efficient, necessitating fine-tuning on as few as 50 images and approximately 1% of its parameters. Finally, we demonstrate that a subtle application of classifier-free guidance can improve the model's performance on depth and normal estimation. We also show that pixel-aligned training, as is characteristic of perception tasks, significantly enhances the model's ability to preserve fine details. DICEPTION offers valuable insights and presents a promising direction for the development of advanced diffusion-based visual generalist models. Code and Model: https://github.com/aim-uofa/Diception