Contextualized Sparse Representations for Real-Time Open-Domain Question Answering

TL;DR

This paper introduces Sparc, a contextualized sparse representation method that enhances phrase embeddings for open-domain question answering, achieving higher accuracy and faster inference than existing models.

Contribution

It proposes a novel sparse vector learning approach using rectified self-attention, improving phrase retrieval accuracy and speed in open-domain QA systems.

Findings

Achieves over 4% improvement on CuratedTREC and SQuAD-Open datasets.

Outperforms previous retrieve & read models in accuracy and inference speed.

Provides a scalable, efficient phrase embedding method for real-time QA.

Abstract

Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (Sparc). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with Sparc, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.