Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation

TL;DR

This paper reveals the semantic structure within diffusion score models and introduces DUSA, a method that leverages these priors for effective test-time adaptation of classifiers and predictors, using minimal diffusion steps.

Contribution

The work uncovers the semantic structure in diffusion scores and proposes DUSA, a novel approach for test-time adaptation that extracts knowledge from a single denoising timestep.

Findings

DUSA effectively adapts pre-trained models across diverse scenarios.

Using a single timestep simplifies likelihood estimation.

Thorough ablation studies highlight key components of DUSA.

Abstract

Capitalizing on the complementary advantages of generative and discriminative models has always been a compelling vision in machine learning, backed by a growing body of research. This work discloses the hidden semantic structure within score-based generative models, unveiling their potential as effective discriminative priors. Inspired by our theoretical findings, we propose DUSA to exploit the structured semantic priors underlying diffusion score to facilitate the test-time adaptation of image classifiers or dense predictors. Notably, DUSA extracts knowledge from a single timestep of denoising diffusion, lifting the curse of Monte Carlo-based likelihood estimation over timesteps. We demonstrate the efficacy of our DUSA in adapting a wide variety of competitive pre-trained discriminative models on diverse test-time scenarios. Additionally, a thorough ablation study is conducted to dissect the pivotal elements in DUSA. Code is publicly available at https://github.com/BIT-DA/DUSA.