Fine-Tuning Transformers: Vocabulary Transfer

TL;DR

This paper investigates how corpus-specific tokenization and vocabulary transfer during fine-tuning of transformer models can enhance transfer learning efficiency and model performance in NLP tasks.

Contribution

It introduces the concept of vocabulary transfer and demonstrates its effectiveness in improving fine-tuning speed and accuracy.

Findings

Vocabulary transfer speeds up fine-tuning.

Corpus-specific tokenization boosts performance.

Vocabulary transfer enhances transfer learning efficiency.

Abstract

Transformers are responsible for the vast majority of recent advances in natural language processing. The majority of practical natural language processing applications of these models are typically enabled through transfer learning. This paper studies if corpus-specific tokenization used for fine-tuning improves the resulting performance of the model. Through a series of experiments, we demonstrate that such tokenization combined with the initialization and fine-tuning strategy for the vocabulary tokens speeds up the transfer and boosts the performance of the fine-tuned model. We call this aspect of transfer facilitation vocabulary transfer.