Flo: a Semantic Foundation for Progressive Stream Processing

TL;DR

Flo introduces a semantic foundation for stream processing that unifies various streaming models through properties ensuring deterministic and fresh outputs, leveraging a type system for bounded and unbounded streams.

Contribution

The paper defines core semantic properties for streaming systems and models them in Flo, a flexible language that captures key ideas from existing systems like Flink and LVars.

Findings

Flo's semantics encompass key streaming systems

Properties guarantee deterministic and fresh outputs

Type system distinguishes bounded and unbounded streams

Abstract

Streaming systems are present throughout modern applications, processing continuous data in real-time. Existing streaming languages have a variety of semantic models and guarantees that are often incompatible. Yet all these languages are considered "streaming" -- what do they have in common? In this paper, we identify two general yet precise semantic properties: streaming progress and eager execution. Together, they ensure that streaming outputs are deterministic and kept fresh with respect to streaming inputs. We formally define these properties in the context of Flo, a parameterized streaming language that abstracts over dataflow operators and the underlying structure of streams. It leverages a lightweight type system to distinguish bounded streams, which allow operators to block on termination, from unbounded ones. Furthermore, Flo provides constructs for dataflow composition and nested graphs with cycles. To demonstrate the generality of our properties, we show how key ideas from representative streaming and incremental computation systems -- Flink, LVars, and DBSP -- have semantics that can be modeled in Flo and guarantees that map to our properties.