One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data

TL;DR

This paper introduces OTA, a universal transfer framework that adapts any Vision Foundation Model to various downstream tasks with minimal data, using a two-step process involving fine-tuning and knowledge distillation, without relying on upstream data.

Contribution

The paper presents a novel universal transfer framework that works across different vision models and tasks with few data, enhancing flexibility and privacy in model deployment.

Findings

OTA outperforms existing transfer methods in few data scenarios.

The framework is independent of upstream data, models, and tasks.

Experiments demonstrate superior effectiveness and generality.

Abstract

The foundation model is not the last chapter of the model production pipeline. Transferring with few data in a general way to thousands of downstream tasks is becoming a trend of the foundation model's application. In this paper, we proposed a universal transfer framework: One to Transfer All (OTA) to transfer any Vision Foundation Model (VFM) to any downstream tasks with few downstream data. We first transfer a VFM to a task-specific model by Image Re-representation Fine-tuning (IRF) then distilling knowledge from a task-specific model to a deployed model with data produced by Downstream Image-Guided Generation (DIGG). OTA has no dependency on upstream data, VFM, and downstream tasks when transferring. It also provides a way for VFM researchers to release their upstream information for better transferring but not leaking data due to privacy requirements. Massive experiments validate the effectiveness and superiority of our methods in few data setting. Our code will be released.