Understanding Knowledge Transferability for Transfer Learning: A Survey

TL;DR

This survey reviews transferability metrics in transfer learning, categorizing them, analyzing their theoretical foundations, and guiding their selection to improve AI system reliability and effectiveness.

Contribution

It provides a unified taxonomy of transferability metrics, analyzes their applicability, and offers insights to guide future research and practical applications in transfer learning.

Findings

Categorizes transferability metrics based on knowledge types and measurement granularity.

Analyzes the theoretical underpinnings of various transferability metrics.

Highlights open challenges and proposes future research directions.

Abstract

Transfer learning has become an essential paradigm in artificial intelligence, enabling the transfer of knowledge from a source task to improve performance on a target task. This approach, particularly through techniques such as pretraining and fine-tuning, has seen significant success in fields like computer vision and natural language processing. However, despite its widespread use, how to reliably assess the transferability of knowledge remains a challenge. Understanding the theoretical underpinnings of each transferability metric is critical for ensuring the success of transfer learning. In this survey, we provide a unified taxonomy of transferability metrics, categorizing them based on transferable knowledge types and measurement granularity. This work examines the various metrics developed to evaluate the potential of source knowledge for transfer learning and their applicability across different learning paradigms emphasizing the need for careful selection of these metrics. By offering insights into how different metrics work under varying conditions, this survey aims to guide researchers and practitioners in selecting the most appropriate metric for specific applications, contributing to more efficient, reliable, and trustworthy AI systems. Finally, we discuss some open challenges in this field and propose future research directions to further advance the application of transferability metrics in trustworthy transfer learning.