A simple method for domain adaptation of sentence embeddings

TL;DR

This paper introduces a simple, universal finetuning method for Google's Universal Sentence Encoder using a Siamese architecture, improving domain adaptation for sentence embeddings across various datasets.

Contribution

A novel, straightforward finetuning approach for USE that enhances domain adaptation and can combine multiple datasets with different annotations.

Findings

Improved performance on domain-specific tasks.

Effective combination of datasets with different annotations.

Comparable or superior results to traditional finetuning methods.

Abstract

Pre-trained sentence embeddings have been shown to be very useful for a variety of NLP tasks. Due to the fact that training such embeddings requires a large amount of data, they are commonly trained on a variety of text data. An adaptation to specific domains could improve results in many cases, but such a finetuning is usually problem-dependent and poses the risk of over-adapting to the data used for adaptation. In this paper, we present a simple universal method for finetuning Google's Universal Sentence Encoder (USE) using a Siamese architecture. We demonstrate how to use this approach for a variety of data sets and present results on different data sets representing similar problems. The approach is also compared to traditional finetuning on these data sets. As a further advantage, the approach can be used for combining data sets with different annotations. We also present an embedding finetuned on all data sets in parallel.