Dynamic Memory Allocation Policies for Postings in Real-Time Twitter Search

TL;DR

This paper investigates dynamic memory allocation policies for in-memory index structures in real-time Twitter search, aiming to optimize query speed and memory usage amid high tweet arrival rates.

Contribution

It introduces a novel dynamic postings allocation policy using increasing slice sizes from fixed memory pools, balancing efficiency and fragmentation.

Findings

The proposed policy improves query evaluation speed.

It achieves better memory utilization compared to static policies.

Analytical models validate the effectiveness of the approach.

Abstract

We explore a real-time Twitter search application where tweets are arriving at a rate of several thousands per second. Real-time search demands that they be indexed and searchable immediately, which leads to a number of implementation challenges. In this paper, we focus on one aspect: dynamic postings allocation policies for index structures that are completely held in main memory. The core issue can be characterized as a "Goldilocks Problem". Because memory remains today a scare resource, an allocation policy that is too aggressive leads to inefficient utilization, while a policy that is too conservative is slow and leads to fragmented postings lists. We present a dynamic postings allocation policy that allocates memory in increasingly-larger "slices" from a small number of large, fixed pools of memory. Through analytical models and experiments, we explore different settings that balance time (query evaluation speed) and space (memory utilization).