The Performance Envelope of Inverted Indexing on Modern Hardware

TL;DR

This study evaluates the performance limits of inverted indexing on modern hardware, revealing that physical media characteristics dominate throughput and that current techniques are nearing physical device constraints.

Contribution

It provides a comprehensive benchmarking of inverted indexing performance across various hardware configurations, highlighting physical media as the primary bottleneck and suggesting the need for rethinking indexing pipelines.

Findings

Physical media characteristics are the main performance determinants.

Isolating source and target media maximizes indexing throughput.

Current indexing techniques are approaching physical device limits.

Abstract

This paper explores the performance envelope of "traditional" inverted indexing on modern hardware using the implementation in the open-source Lucene search library. We benchmark indexing throughput on a single high-end multi-core commodity server in a number of configurations varying the media of the source collection and target index, examining a network-attached store, a direct-attached disk array, and an SSD. Experiments show that the largest determinants of performance are the physical characteristics of the source and target media, and that physically isolating the two yields the highest indexing throughput. Results suggest that current indexing techniques have reached physical device limits, and that further algorithmic improvements in performance are unlikely without rethinking the inverted indexing pipeline in light of observed bottlenecks.