How Transferable are Neural Networks in NLP Applications?

TL;DR

This paper systematically investigates how well neural networks transfer knowledge across NLP tasks, highlighting the variability and factors affecting transferability in neural NLP models.

Contribution

It provides a comprehensive analysis of neural network transferability in NLP, addressing gaps from prior casual applications and inconsistent conclusions.

Findings

Transferability varies significantly across NLP tasks.

Certain model architectures transfer better than others.

Transfer learning effectiveness depends on domain similarity.

Abstract

Transfer learning is aimed to make use of valuable knowledge in a source domain to help model performance in a target domain. It is particularly important to neural networks, which are very likely to be overfitting. In some fields like image processing, many studies have shown the effectiveness of neural network-based transfer learning. For neural NLP, however, existing studies have only casually applied transfer learning, and conclusions are inconsistent. In this paper, we conduct systematic case studies and provide an illuminating picture on the transferability of neural networks in NLP.