Efficient Domain Adaptation of Sentence Embeddings Using Adapters

TL;DR

This paper introduces a resource-efficient method for domain adaptation of sentence embeddings by training lightweight adapters, achieving near state-of-the-art performance with significantly fewer trained parameters.

Contribution

The authors propose using domain-specific adapters for sentence embeddings, reducing resource requirements compared to full fine-tuning while maintaining high performance.

Findings

Adapters achieve within 1% of fully fine-tuned models.

Only 3.6% of parameters need training with adapters.

Method is resource-efficient and maintains competitive accuracy.

Abstract

Sentence embeddings enable us to capture the semantic similarity of short texts. Most sentence embedding models are trained for general semantic textual similarity tasks. Therefore, to use sentence embeddings in a particular domain, the model must be adapted to it in order to achieve good results. Usually, this is done by fine-tuning the entire sentence embedding model for the domain of interest. While this approach yields state-of-the-art results, all of the model's weights are updated during fine-tuning, making this method resource-intensive. Therefore, instead of fine-tuning entire sentence embedding models for each target domain individually, we propose to train lightweight adapters. These domain-specific adapters do not require fine-tuning all underlying sentence embedding model parameters. Instead, we only train a small number of additional parameters while keeping the weights of the underlying sentence embedding model fixed. Training domain-specific adapters allows always using the same base model and only exchanging the domain-specific adapters to adapt sentence embeddings to a specific domain. We show that using adapters for parameter-efficient domain adaptation of sentence embeddings yields competitive performance within 1% of a domain-adapted, entirely fine-tuned sentence embedding model while only training approximately 3.6% of the parameters.