A Review of Deep Transfer Learning and Recent Advancements

TL;DR

This paper reviews deep transfer learning, highlighting its methods, recent applications like Covid-19 detection, limitations such as catastrophic forgetting, and future research directions to improve its effectiveness and applicability.

Contribution

It provides a comprehensive overview of DTL definitions, taxonomy, recent techniques, experimental analyses, and discusses limitations and future research trends.

Findings

DTL enables high-accuracy Covid-19 detection with minimal data

Reduces training costs, making DTL suitable for edge devices

Identifies key limitations like catastrophic forgetting and biased models

Abstract

Deep learning has been the answer to many machine learning problems during the past two decades. However, it comes with two major constraints: dependency on extensive labeled data and training costs. Transfer learning in deep learning, known as Deep Transfer Learning (DTL), attempts to reduce such dependency and costs by reusing an obtained knowledge from a source data/task in training on a target data/task. Most applied DTL techniques are network/model-based approaches. These methods reduce the dependency of deep learning models on extensive training data and drastically decrease training costs. As a result, researchers detected Covid-19 infection on chest X-Rays with high accuracy at the beginning of the pandemic with minimal data using DTL techniques. Also, the training cost reduction makes DTL viable on edge devices with limited resources. Like any new advancement, DTL methods have their own limitations, and a successful transfer depends on some adjustments for different scenarios. In this paper, we review the definition and taxonomy of deep transfer learning and well-known methods. Then we investigate the DTL approaches by reviewing recent applied DTL techniques in the past five years. Further, we review some experimental analyses of DTLs to learn the best practice for applying DTL in different scenarios. Moreover, the limitations of DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are discussed, along with possible solutions and research trends.