Sequences, yet Functions: The Dual Nature of Data-Stream Processing

TL;DR

This paper proposes a novel functional programming-based model for data-stream processing, viewing streams as functions rather than sequences, which enhances reasoning about query semantics and provides a solid foundation for future query languages.

Contribution

It introduces a time-slicing semantics model for data-streams, demonstrating that streams are monads, and links stream processing concepts to functional programming and mathematics.

Findings

Data-streams modeled as functions improve query reasoning.

Streams are proven to be monads, enabling functional programming techniques.

The model provides a foundation for future data-stream query languages.

Abstract

Data-stream processing has continuously risen in importance as the amount of available data has been steadily increas- ing over the last decade. Besides traditional domains such as data-center monitoring and click analytics, there is an increasing number of network-enabled production machines that generate continuous streams of data. Due to their continuous nature, queries on data-streams can be more complex, and distinctly harder to understand then database queries. As users have to consider operational details, maintenance and debugging become challenging. Current approaches model data-streams as sequences, be- cause this is the way they are physically received. These models result in an implementation-focused perspective. We explore an alternate way of modeling data-streams by focusing on time-slicing semantics. This focus results in a model based on functions, which is better suited for reasoning about query semantics. By adapting the definitions of relevant concepts in stream processing to our model, we illustrate the practical useful- ness of our approach. Thereby, we link data-streams and query primitives to concepts in functional programming and mathematics. Most noteworthy, we prove that data-streams are monads, and show how to derive monad definitions for current data-stream models. We provide an abstract, yet practical perspective on data- stream related subjects based on a sound, consistent query model. Our work can serve as solid foundation for future data-stream query-languages.