Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models

TL;DR

This paper introduces Hub-Pathway, a framework for transfer learning from a diverse model hub, enabling efficient knowledge transfer and improved performance across tasks by dynamically selecting and aggregating models.

Contribution

It proposes a novel end-to-end trainable framework that dynamically explores and exploits knowledge from multiple pre-trained models in a model hub for transfer learning.

Findings

Achieves state-of-the-art results on vision and reinforcement learning tasks.

Effectively explores diverse pathways to optimize knowledge transfer.

Enhances model performance by data-dependent pathway routing.

Abstract

Transfer learning aims to leverage knowledge from pre-trained models to benefit the target task. Prior transfer learning work mainly transfers from a single model. However, with the emergence of deep models pre-trained from different resources, model hubs consisting of diverse models with various architectures, pre-trained datasets and learning paradigms are available. Directly applying single-model transfer learning methods to each model wastes the abundant knowledge of the model hub and suffers from high computational cost. In this paper, we propose a Hub-Pathway framework to enable knowledge transfer from a model hub. The framework generates data-dependent pathway weights, based on which we assign the pathway routes at the input level to decide which pre-trained models are activated and passed through, and then set the pathway aggregation at the output level to aggregate the knowledge from different models to make predictions. The proposed framework can be trained end-to-end with the target task-specific loss, where it learns to explore better pathway configurations and exploit the knowledge in pre-trained models for each target datum. We utilize a noisy pathway generator and design an exploration loss to further explore different pathways throughout the model hub. To fully exploit the knowledge in pre-trained models, each model is further trained by specific data that activate it, which ensures its performance and enhances knowledge transfer. Experiment results on computer vision and reinforcement learning tasks demonstrate that the proposed Hub-Pathway framework achieves the state-of-the-art performance for model hub transfer learning.