Flexible retrieval with NMSLIB and FlexNeuART

TL;DR

This paper introduces FlexNeuART, a flexible retrieval toolkit that extends NMSLIB for efficient mixed dense and sparse representation retrieval, supporting various distance functions and integration with neural and classic ranking signals.

Contribution

It presents a modular, extendible retrieval system that integrates NMSLIB with new capabilities for mixed representation retrieval in IR and QA tasks.

Findings

Supports efficient retrieval of mixed dense and sparse representations.

Extends NMSLIB with new distance functions for flexible similarity search.

Enables integration of neural and classic ranking signals in retrieval pipelines.

Abstract

Our objective is to introduce to the NLP community an existing k-NN search library NMSLIB, a new retrieval toolkit FlexNeuART, as well as their integration capabilities. NMSLIB, while being one the fastest k-NN search libraries, is quite generic and supports a variety of distance/similarity functions. Because the library relies on the distance-based structure-agnostic algorithms, it can be further extended by adding new distances. FlexNeuART is a modular, extendible and flexible toolkit for candidate generation in IR and QA applications, which supports mixing of classic and neural ranking signals. FlexNeuART can efficiently retrieve mixed dense and sparse representations (with weights learned from training data), which is achieved by extending NMSLIB. In that, other retrieval systems work with purely sparse representations (e.g., Lucene), purely dense representations (e.g., FAISS and Annoy), or only perform mixing at the re-ranking stage.