Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering

TL;DR

This paper introduces TASER, a task-aware model for dense retrieval in open-domain question answering that shares parameters to improve efficiency and robustness, outperforming traditional bi-encoder models and BM25.

Contribution

TASER enables parameter sharing in dense retrieval models by interleaving shared and specialized blocks, improving efficiency and robustness over existing bi-encoder architectures.

Findings

TASER surpasses BM25 in accuracy on five QA datasets.

TASER uses about 60% of the parameters of bi-encoder retrievers.

TASER demonstrates greater robustness in out-of-domain evaluations.

Abstract

Given its effectiveness on knowledge-intensive natural language processing tasks, dense retrieval models have become increasingly popular. Specifically, the de-facto architecture for open-domain question answering uses two isomorphic encoders that are initialized from the same pretrained model but separately parameterized for questions and passages. This bi-encoder architecture is parameter-inefficient in that there is no parameter sharing between encoders. Further, recent studies show that such dense retrievers underperform BM25 in various settings. We thus propose a new architecture, Task-aware Specialization for dense Retrieval (TASER), which enables parameter sharing by interleaving shared and specialized blocks in a single encoder. Our experiments on five question answering datasets show that TASER can achieve superior accuracy, surpassing BM25, while using about 60% of the parameters as bi-encoder dense retrievers. In out-of-domain evaluations, TASER is also empirically more robust than bi-encoder dense retrievers. Our code is available at https://github.com/microsoft/taser.