Downstream: efficient cross-platform algorithms for fixed-capacity stream downsampling

TL;DR

The paper introduces the Downstream library, which provides efficient algorithms for various fixed-capacity stream downsampling methods, supporting multiple platforms and programming languages for real-time data stream summarization.

Contribution

It presents novel algorithms for steady, stretched, and tilted downsampling, supporting diverse application needs while maintaining efficiency and memory optimality.

Findings

Supports multiple downsampling strategies with high efficiency

Enables cross-platform and multi-language implementation

Provides extensive testing and documentation for broad usability

Abstract

Due to ongoing accrual over long durations, a defining characteristic of real-world data streams is the requirement for rolling, often real-time, mechanisms to coarsen or summarize stream history. One common data structure for this purpose is the ring buffer, which maintains a running downsample comprising most recent stream data. In some downsampling scenarios, however, it can instead be necessary to maintain data items spanning the entirety of elapsed stream history. Fortunately, approaches generalizing the ring buffer mechanism have been devised to support alternate downsample compositions, while maintaining the ring buffer's update efficiency and optimal use of memory capacity. The Downstream library implements algorithms supporting three such downsampling generalizations: (1) "steady," which curates data evenly spaced across the stream history; (2) "stretched," which prioritizes older data; and (3) "tilted," which prioritizes recent data. To enable a broad spectrum of applications ranging from embedded devices to high-performance computing nodes and AI/ML hardware accelerators, Downstream supports multiple programming languages, including C++, Rust, Python, Zig, and the Cerebras Software Language. For seamless interoperation, the library incorporates distribution through multiple packaging frameworks, extensive cross-implementation testing, and cross-implementation documentation.