Towards Efficient Task-Driven Model Reprogramming with Foundation Models

TL;DR

This paper introduces a Task-Driven Model Reprogramming framework that enables efficient knowledge transfer from large vision foundation models to smaller downstream models, addressing domain mismatch and limited data issues.

Contribution

The proposed TDMR framework reprograms foundation models into a proxy space and uses progressive distillation, allowing effective transfer to various target models with limited data.

Findings

TDMR improves transfer efficiency across CNN and transformer models.

The method outperforms traditional fine-tuning and knowledge distillation approaches.

TDMR is compatible with different model architectures and limited target data.

Abstract

Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. Moreover, the data used for pretraining foundation models are usually invisible and very different from the target data of downstream tasks. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task that has a quite different architecture with only downstream target data. Existing transfer learning or knowledge distillation methods depend on either the same model structure or finetuning of the foundation model. Thus, naively introducing these methods can be either infeasible or very inefficient. To address this, we propose a Task-Driven Model Reprogramming (TDMR) framework. Specifically, we reprogram the foundation model to project the knowledge into a proxy space, which alleviates the adverse effect of task mismatch and domain inconsistency. Then, we reprogram the target model via progressive distillation from the proxy space to efficiently learn the knowledge from the reprogrammed foundation model. TDMR is compatible with different pre-trained model types (CNN, transformer or their mix) and limited target data, and promotes the wide applications of vision foundation models to downstream tasks in a cost-effective manner. Extensive experiments on different downstream classification tasks and target model structures demonstrate the effectiveness of our methods with both CNNs and transformer foundation models.