Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Models

TL;DR

This paper introduces Bayesian Power Steering (BPS), a novel Bayesian fine-tuning method for large diffusion models that effectively adapts pre-trained models to new tasks with limited data, outperforming existing approaches.

Contribution

The paper presents BPS, a new Bayesian framework for fine-tuning diffusion models by extracting task-specific knowledge from prior distributions, with a unique network structure and intervention strategy.

Findings

BPS outperforms contemporary methods across various tasks.

Achieves an FID score of 10.49 on COCO17 with limited data.

Efficiently leverages large diffusion models for domain adaptation.

Abstract

We propose a Bayesian framework for fine-tuning large diffusion models with a novel network structure called Bayesian Power Steering (BPS). We clarify the meaning behind adaptation from a \textit{large probability space} to a \textit{small probability space} and explore the task of fine-tuning pre-trained models using learnable modules from a Bayesian perspective. BPS extracts task-specific knowledge from a pre-trained model's learned prior distribution. It efficiently leverages large diffusion models, differentially intervening different hidden features with a head-heavy and foot-light configuration. Experiments highlight the superiority of BPS over contemporary methods across a range of tasks even with limited amount of data. Notably, BPS attains an FID score of 10.49 under the sketch condition on the COCO17 dataset.