Relevance Filtering for Embedding-based Retrieval

TL;DR

This paper proposes a relevance filtering method for embedding-based retrieval that improves precision by mapping cosine similarity scores to interpretable scores and applying a global threshold, validated on datasets and real-world e-commerce data.

Contribution

Introduces the Cosine Adapter, a novel component that enhances filtering in embedding-based retrieval by mapping similarity scores to interpretable scores for better relevance filtering.

Findings

Significantly increased retrieval precision on MS MARCO and Walmart datasets.

Effective in real-world e-commerce search, validated through online A/B testing.

Small recall loss but improved overall search quality.

Abstract

In embedding-based retrieval, Approximate Nearest Neighbor (ANN) search enables efficient retrieval of similar items from large-scale datasets. While maximizing recall of relevant items is usually the goal of retrieval systems, a low precision may lead to a poor search experience. Unlike lexical retrieval, which inherently limits the size of the retrieved set through keyword matching, dense retrieval via ANN search has no natural cutoff. Moreover, the cosine similarity scores of embedding vectors are often optimized via contrastive or ranking losses, which make them difficult to interpret. Consequently, relying on top-K or cosine-similarity cutoff is often insufficient to filter out irrelevant results effectively. This issue is prominent in product search, where the number of relevant products is often small. This paper introduces a novel relevance filtering component (called "Cosine Adapter") for embedding-based retrieval to address this challenge. Our approach maps raw cosine similarity scores to interpretable scores using a query-dependent mapping function. We then apply a global threshold on the mapped scores to filter out irrelevant results. We are able to significantly increase the precision of the retrieved set, at the expense of a small loss of recall. The effectiveness of our approach is demonstrated through experiments on both public MS MARCO dataset and internal Walmart product search data. Furthermore, online A/B testing on the Walmart site validates the practical value of our approach in real-world e-commerce settings.