CEQE: Contextualized Embeddings for Query Expansion

TL;DR

This paper introduces CEQE, a novel query expansion method using contextualized embeddings like BERT, which significantly improves retrieval effectiveness over static models and traditional PRF methods.

Contribution

The paper proposes CEQE, a new model leveraging query-focused contextualized embeddings for query expansion, demonstrating superior performance in ad-hoc retrieval tasks.

Findings

CEQE outperforms static embedding methods by up to 18% on Robust and 31% on Deep Learning.

CEQE improves over traditional PRF models in multiple retrieval scenarios.

Multiple expansion and reranking passes further enhance retrieval effectiveness.

Abstract

In this work we leverage recent advances in context-sensitive language models to improve the task of query expansion. Contextualized word representation models, such as ELMo and BERT, are rapidly replacing static embedding models. We propose a new model, Contextualized Embeddings for Query Expansion (CEQE), that utilizes query-focused contextualized embedding vectors. We study the behavior of contextual representations generated for query expansion in ad-hoc document retrieval. We conduct our experiments on probabilistic retrieval models as well as in combination with neural ranking models. We evaluate CEQE on two standard TREC collections: Robust and Deep Learning. We find that CEQE outperforms static embedding-based expansion methods on multiple collections (by up to 18% on Robust and 31% on Deep Learning on average precision) and also improves over proven probabilistic pseudo-relevance feedback (PRF) models. We further find that multiple passes of expansion and reranking result in continued gains in effectiveness with CEQE-based approaches outperforming other approaches. The final model incorporating neural and CEQE-based expansion score achieves gains of up to 5% in P@20 and 2% in AP on Robust over the state-of-the-art transformer-based re-ranking model, Birch.