AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain

TL;DR

This paper introduces AVocaDo, a method to adapt and optimize the vocabulary during transfer learning by expanding it with domain-specific words and regularizing embeddings, leading to improved performance across various domains.

Contribution

It proposes treating vocabulary as an optimizable parameter and updating it with domain-specific words while preserving pretrained embeddings through regularization.

Findings

Consistent performance improvements across biomedical, computer science, news, and review domains.

Effective vocabulary adaptation enhances downstream task performance.

Regularization prevents overfitting of new vocabulary embeddings.

Abstract

During the fine-tuning phase of transfer learning, the pretrained vocabulary remains unchanged, while model parameters are updated. The vocabulary generated based on the pretrained data is suboptimal for downstream data when domain discrepancy exists. We propose to consider the vocabulary as an optimizable parameter, allowing us to update the vocabulary by expanding it with domain-specific vocabulary based on a tokenization statistic. Furthermore, we preserve the embeddings of the added words from overfitting to downstream data by utilizing knowledge learned from a pretrained language model with a regularization term. Our method achieved consistent performance improvements on diverse domains (i.e., biomedical, computer science, news, and reviews).