TADA: Efficient Task-Agnostic Domain Adaptation for Transformers

TL;DR

TADA is a modular, parameter-efficient domain adaptation method for transformers that retrains embeddings and tokenizers, enabling effective multi-domain adaptation without additional parameters or complex training.

Contribution

Introduces TADA, a novel task-agnostic domain adaptation approach that retrains embeddings and tokenizers while freezing other parameters, improving efficiency and effectiveness.

Findings

TADA performs well across 14 domains and 4 downstream tasks.

It is more efficient than full pre-training and adapters.

TADA requires no additional parameters or complex training steps.

Abstract

Intermediate training of pre-trained transformer-based language models on domain-specific data leads to substantial gains for downstream tasks. To increase efficiency and prevent catastrophic forgetting alleviated from full domain-adaptive pre-training, approaches such as adapters have been developed. However, these require additional parameters for each layer, and are criticized for their limited expressiveness. In this work, we introduce TADA, a novel task-agnostic domain adaptation method which is modular, parameter-efficient, and thus, data-efficient. Within TADA, we retrain the embeddings to learn domain-aware input representations and tokenizers for the transformer encoder, while freezing all other parameters of the model. Then, task-specific fine-tuning is performed. We further conduct experiments with meta-embeddings and newly introduced meta-tokenizers, resulting in one model per task in multi-domain use cases. Our broad evaluation in 4 downstream tasks for 14 domains across single- and multi-domain setups and high- and low-resource scenarios reveals that TADA is an effective and efficient alternative to full domain-adaptive pre-training and adapters for domain adaptation, while not introducing additional parameters or complex training steps.