Vocabulary Transfer for Biomedical Texts: Add Tokens if You Can Not Add Data

TL;DR

This paper explores vocabulary transfer and extension techniques to improve biomedical NLP models, addressing data scarcity and privacy issues, and demonstrating measurable performance and efficiency gains.

Contribution

It introduces vocabulary extension as a method to incorporate domain-specific biomedical terms, enhancing model performance in data-limited biomedical NLP tasks.

Findings

Vocabulary extension improves downstream performance.

Vocabulary transfer reduces inference time.

Enhancements are significant in privacy-constrained settings.

Abstract

Working within specific NLP subdomains presents significant challenges, primarily due to a persistent deficit of data. Stringent privacy concerns and limited data accessibility often drive this shortage. Additionally, the medical domain demands high accuracy, where even marginal improvements in model performance can have profound impacts. In this study, we investigate the potential of vocabulary transfer to enhance model performance in biomedical NLP tasks. Specifically, we focus on vocabulary extension, a technique that involves expanding the target vocabulary to incorporate domain-specific biomedical terms. Our findings demonstrate that vocabulary extension, leads to measurable improvements in both downstream model performance and inference time.