Task-oriented Learnable Diffusion Timesteps for Universal Few-shot Learning of Dense Tasks

TL;DR

This paper introduces a novel framework that adaptively selects and consolidates diffusion timesteps for improved few-shot dense prediction tasks, enhancing performance on unseen tasks with minimal data.

Contribution

It proposes two modules, TTS and TFC, for adaptive timestep selection and consolidation, advancing diffusion models for universal few-shot dense learning.

Findings

Achieves superior dense prediction performance with few support queries.

Validates effectiveness on large-scale Taskonomy dataset.

Enhances diffusion models for universal few-shot learning.

Abstract

Denoising diffusion probabilistic models have brought tremendous advances in generative tasks, achieving state-of-the-art performance thus far. Current diffusion model-based applications exploit the power of learned visual representations from multistep forward-backward Markovian processes for single-task prediction tasks by attaching a task-specific decoder. However, the heuristic selection of diffusion timestep features still heavily relies on empirical intuition, often leading to sub-optimal performance biased towards certain tasks. To alleviate this constraint, we investigate the significance of versatile diffusion timestep features by adaptively selecting timesteps best suited for the few-shot dense prediction task, evaluated on an arbitrary unseen task. To this end, we propose two modules: Task-aware Timestep Selection (TTS) to select ideal diffusion timesteps based on timestep-wise losses and similarity scores, and Timestep Feature Consolidation (TFC) to consolidate the selected timestep features to improve the dense predictive performance in a few-shot setting. Accompanied by our parameter-efficient fine-tuning adapter, our framework effectively achieves superiority in dense prediction performance given only a few support queries. We empirically validate our learnable timestep consolidation method on the large-scale challenging Taskonomy dataset for dense prediction, particularly for practical universal and few-shot learning scenarios.