Counter Pools: Counter Representation for Efficient Stream Processing

TL;DR

This paper introduces a memory-efficient encoding scheme for counters in stream processing systems, optimizing counter sizes within fixed memory pools to improve performance and reduce memory usage.

Contribution

It proposes a novel counter encoding method that dynamically sizes counters within fixed memory pools, enhancing efficiency in stream processing.

Findings

Significant memory savings in stream algorithms

Improved performance across various workloads

Effective counter management within fixed memory pools

Abstract

Due to the large data volume and number of distinct elements, space is often the bottleneck of many stream processing systems. The data structures used by these systems often consist of counters whose optimization yields significant memory savings. The challenge lies in balancing the size of the counters: too small, and they overflow; too large, and memory capacity limits their number. In this work, we suggest an efficient encoding scheme that sizes each counter according to its needs. Our approach uses fixed-sized pools of memory (e.g., a single memory word or 64 bits), where each pool manages a small number of counters. We pay special attention to performance and demonstrate considerable improvements for various streaming algorithms and workload characteristics.