Unleashing Foundation Vision Models: Adaptive Transfer for Diverse Data-Limited Scientific Domains

TL;DR

This paper introduces CLAdapter, a novel attention-based method that adapts large-scale pre-trained vision models to various data-limited scientific domains, achieving state-of-the-art results.

Contribution

The paper presents CLAdapter, a unified, attention-based adapter that effectively transfers knowledge from large-scale pre-trained models to diverse, data-scarce scientific tasks.

Findings

Achieves state-of-the-art performance on 10 diverse datasets.

Effectively adapts models across multiple architectures and data types.

Demonstrates significant improvements in data-limited scenarios.

Abstract

In the big data era, the computer vision field benefits from large-scale datasets such as LAION-2B, LAION-400M, and ImageNet-21K, Kinetics, on which popular models like the ViT and ConvNeXt series have been pre-trained, acquiring substantial knowledge. However, numerous downstream tasks in specialized and data-limited scientific domains continue to pose significant challenges. In this paper, we propose a novel Cluster Attention Adapter (CLAdapter), which refines and adapts the rich representations learned from large-scale data to various data-limited downstream tasks. Specifically, CLAdapter introduces attention mechanisms and cluster centers to personalize the enhancement of transformed features through distribution correlation and transformation matrices. This enables models fine-tuned with CLAdapter to learn distinct representations tailored to different feature sets, facilitating the models' adaptation from rich pre-trained features to various downstream scenarios effectively. In addition, CLAdapter's unified interface design allows for seamless integration with multiple model architectures, including CNNs and Transformers, in both 2D and 3D contexts. Through extensive experiments on 10 datasets spanning domains such as generic, multimedia, biological, medical, industrial, agricultural, environmental, geographical, materials science, out-of-distribution (OOD), and 3D analysis, CLAdapter achieves state-of-the-art performance across diverse data-limited scientific domains, demonstrating its effectiveness in unleashing the potential of foundation vision models via adaptive transfer. Code is available at https://github.com/qklee-lz/CLAdapter.