Integrating the Probabilistic Models BM25/BM25F into Lucene

TL;DR

This paper details the implementation of BM25 and BM25F probabilistic retrieval models within the Lucene framework, highlighting their state-of-the-art performance in information retrieval tasks for unstructured and structured documents.

Contribution

It introduces the integration of BM25 and BM25F models into Lucene, enabling improved retrieval performance for both plain text and structured documents.

Findings

BM25 and BM25F are effective state-of-the-art IR models.

Implementation within Lucene facilitates practical use in search applications.

Models perform well on TREC benchmarks.

Abstract

This document describes the BM25 and BM25F implementation using the Lucene Java Framework. Both models have stood out at TREC by their performance and are considered as state-of-the-art in the IR community. BM25 is applied to retrieval on plain text documents, that is for documents that do not contain fields, while BM25F is applied to documents with structure.