Low-Latency Stateful Stream Processing through Timely and Accurate Prefetching

TL;DR

This paper introduces Keyed Prefetching and Timestamp-Aware Caching to reduce latency in stateful stream processing by proactively staging state in memory, enabling overlap of I/O and computation for real-time applications.

Contribution

It presents a novel prefetching technique that decouples data path from state access, significantly lowering latency in large-state stream processing systems.

Findings

Latency reduced for long-running queries

Overlapped I/O with computation effectively

Maintained high throughput

Abstract

Mission-critical applications often run "forever" and process large data volumes in real time while demanding low latency. To handle the large state of these applications, modern streaming engines rely on key-value stores and store state on local storage or remotely, but accessing such state inflates latency. As today's engines tightly couple the data path with state I/O, a tuple triggers state access only when it reaches a stateful operator, placing I/O on the critical path and stalling the CPU. However, the keys used to access the state are frequently known earlier in the query plan. Building on this insight, we propose Keyed Prefetching, which decouples the data path from state access by extracting future access keys at upstream operators and proactively staging the corresponding state in memory before tuples arrive. This overlaps I/O with ongoing computation and hides the latency of large-state accesses. We pair Keyed Prefetching with Timestamp-Aware Caching, a cache-eviction policy that jointly manages previously accessed and prefetched entries to use memory efficiently. Together, these techniques reduce latency for long-running, real-time queries without sacrificing throughput.