Dual-View Distilled BERT for Sentence Embedding

TL;DR

This paper introduces DvBERT, a dual-view approach that enhances sentence embeddings by combining a Siamese view with an interaction view, significantly improving performance on sentence similarity tasks.

Contribution

The paper proposes a novel dual-view framework for BERT-based sentence embeddings, integrating cross-sentence interaction to boost semantic representation quality.

Findings

Outperforms state-of-the-art on six STS tasks

Effectively captures global semantic information

Enhances sentence embedding quality through dual views

Abstract

Recently, BERT realized significant progress for sentence matching via word-level cross sentence attention. However, the performance significantly drops when using siamese BERT-networks to derive two sentence embeddings, which fall short in capturing the global semantic since the word-level attention between two sentences is absent. In this paper, we propose a Dual-view distilled BERT~(DvBERT) for sentence matching with sentence embeddings. Our method deals with a sentence pair from two distinct views, i.e., Siamese View and Interaction View. Siamese View is the backbone where we generate sentence embeddings. Interaction View integrates the cross sentence interaction as multiple teachers to boost the representation ability of sentence embeddings. Experiments on six STS tasks show that our method outperforms the state-of-the-art sentence embedding methods significantly.