Mixture of Physical Priors Adapter for Parameter-Efficient Fine-Tuning

TL;DR

This paper introduces MoPPA, a novel parameter-efficient fine-tuning method that models network weights using a mixture of physical priors derived from fundamental equations, improving accuracy in vision tasks.

Contribution

It proposes the Mixture of Physical Priors Adapter (MoPPA), a lightweight, plug-and-play module that combines heat diffusion, wave propagation, and Poisson's equations for enhanced model adaptation.

Findings

MoPPA improves PEFT accuracy by up to 2.1% on VTAB-1K.

It effectively integrates into transformer architectures.

Demonstrates adaptability across various vision backbones.

Abstract

Most parameter-efficient fine-tuning (PEFT) methods rely on low-rank representations to adapt models. However, these approaches often oversimplify representations, particularly when the underlying data has high-rank or high-frequency components. This limitation hinders the model's ability to capture complex data interactions effectively. In this paper, we propose a novel approach that models network weights by leveraging a combination of physical priors, enabling more accurate approximations. We use three foundational equations -- heat diffusion, wave propagation, and Poisson's steady-state equation -- each contributing distinctive modeling properties: heat diffusion enforces local smoothness, wave propagation facilitates long-range interactions, and Poisson's equation captures global equilibrium. To combine these priors effectively, we introduce the Mixture of Physical Priors Adapter (MoPPA), using an efficient Discrete Cosine Transform (DCT) implementation. To dynamically balance these priors, a route regularization mechanism is designed to adaptively tune their contributions. MoPPA serves as a lightweight, plug-and-play module that seamlessly integrates into transformer architectures, with adaptable complexity depending on the local context. Specifically, using MAE pre-trained ViT-B, MoPPA improves PEFT accuracy by up to 2.1% on VTAB-1K image classification with a comparable number of trainable parameters, and advantages are further validated through experiments across various vision backbones, showcasing MoPPA's effectiveness and adaptability. The code will be made public available.