Unsupervised Transfer Learning via BERT Neuron Selection

TL;DR

This paper introduces a neuron selection method for unsupervised transfer learning with BERT, enabling effective domain adaptation by identifying task-specific neurons and creating fingerprints for source-target similarity assessment.

Contribution

The paper proposes a novel neuron selection algorithm for unsupervised transfer learning with BERT, including multi-source transfer and task-specific fingerprinting, improving transferability and interpretability.

Findings

Higher similarity in task-specific fingerprints correlates with better transfer performance.

Using selected neurons achieves comparable or better results than full model fine-tuning.

The method enhances transfer learning efficiency with fewer task-specific neurons.

Abstract

Recent advancements in language representation models such as BERT have led to a rapid improvement in numerous natural language processing tasks. However, language models usually consist of a few hundred million trainable parameters with embedding space distributed across multiple layers, thus making them challenging to be fine-tuned for a specific task or to be transferred to a new domain. To determine whether there are task-specific neurons that can be exploited for unsupervised transfer learning, we introduce a method for selecting the most important neurons to solve a specific classification task. This algorithm is further extended to multi-source transfer learning by computing the importance of neurons for several single-source transfer learning scenarios between different subsets of data sources. Besides, a task-specific fingerprint for each data source is obtained based on the percentage of the selected neurons in each layer. We perform extensive experiments in unsupervised transfer learning for sentiment analysis, natural language inference and sentence similarity, and compare our results with the existing literature and baselines. Significantly, we found that the source and target data sources with higher degrees of similarity between their task-specific fingerprints demonstrate a better transferability property. We conclude that our method can lead to better performance using just a few hundred task-specific and interpretable neurons.