Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning

TL;DR

This paper introduces a semi-Siamese bi-encoder neural ranking model that employs lightweight fine-tuning techniques to enhance BERT-based bi-encoders' performance efficiently across multiple datasets.

Contribution

It proposes a novel semi-Siamese architecture combined with lightweight fine-tuning methods, improving BERT-based bi-encoder ranking models' effectiveness and efficiency.

Findings

Lightweight fine-tuning improves bi-encoder performance.

Semi-Siamese models outperform traditional bi-encoders.

Methods are effective across multiple datasets.

Abstract

A BERT-based Neural Ranking Model (NRM) can be either a crossencoder or a bi-encoder. Between the two, bi-encoder is highly efficient because all the documents can be pre-processed before the actual query time. In this work, we show two approaches for improving the performance of BERT-based bi-encoders. The first approach is to replace the full fine-tuning step with a lightweight fine-tuning. We examine lightweight fine-tuning methods that are adapter-based, prompt-based, and hybrid of the two. The second approach is to develop semi-Siamese models where queries and documents are handled with a limited amount of difference. The limited difference is realized by learning two lightweight fine-tuning modules, where the main language model of BERT is kept common for both query and document. We provide extensive experiment results for monoBERT, TwinBERT, and ColBERT where three performance metrics are evaluated over Robust04, ClueWeb09b, and MS-MARCO datasets. The results confirm that both lightweight fine-tuning and semi-Siamese are considerably helpful for improving BERT-based bi-encoders. In fact, lightweight fine-tuning is helpful for crossencoder, too