Investigating the Impact of Data Volume and Domain Similarity on Transfer Learning Applications

TL;DR

This paper quantitatively examines how target dataset size and domain similarity influence transfer learning performance, revealing that more data generally improves results and domain similarity affects the choice of fine tuning versus feature extraction.

Contribution

It provides the first detailed analysis of how data volume and domain similarity impact transfer learning effectiveness, guiding better model development strategies.

Findings

Performance improves linearly with log of data size

More data is needed as domain similarity decreases

Fine tuning outperforms feature extraction when domains differ significantly

Abstract

Transfer learning allows practitioners to recognize and apply knowledge learned in previous tasks (source task) to new tasks or new domains (target task), which share some commonality. The two important factors impacting the performance of transfer learning models are: (a) the size of the target dataset, and (b) the similarity in distribution between source and target domains. Thus far, there has been little investigation into just how important these factors are. In this paper, we investigate the impact of target dataset size and source/target domain similarity on model performance through a series of experiments. We find that more data is always beneficial, and model performance improves linearly with the log of data size, until we are out of data. As source/target domains differ, more data is required and fine tuning will render better performance than feature extraction. When source/target domains are similar and data size is small, fine tuning and feature extraction renders equivalent performance. Our hope is that by beginning this quantitative investigation on the effect of data volume and domain similarity in transfer learning we might inspire others to explore the significance of data in developing more accurate statistical models.