On Using Non-Volatile Memory in Apache Lucene

TL;DR

This paper investigates the impact of non-volatile memory (NVDIMM) on Apache Lucene's performance, showing modest improvements and suggesting redesigns for better utilization of byte-addressable NVM.

Contribution

It is the first study to empirically evaluate NVDIMM's effect on Lucene's performance and proposes redesign strategies for optimal NVM integration.

Findings

Modest performance improvements with NVDIMM usage.

Accessing NVM via file system limits potential benefits.

Redesigning Lucene for byte-addressable NVM could yield larger gains.

Abstract

Apache Lucene is a widely popular information retrieval library used to provide search functionality in an extremely wide variety of applications. Naturally, it has to efficiently index and search large number of documents. With non-volatile memory in DIMM form factor (NVDIMM), software now has access to durable, byte-addressable memory with write latency within an order of magnitude of DRAM write latency. In this preliminary article, we present the first reported work on the impact of using NVDIMM on the performance of committing, searching, and near-real time searching in Apache Lucene. We show modest improvements by using NVM but, our empirical study suggests that bigger impact requires redesigning Lucene to access NVM as byte-addressable memory using loads and stores, instead of accessing NVM via the file system.