CircuitFlow: A Domain Specific Language for Dataflow Programming (with appendices)

TL;DR

CircuitFlow is a Haskell-based domain-specific language for dataflow programming that ensures type safety, improves error detection, and scales better than existing tools like Luigi, with strong mathematical foundations and efficient performance.

Contribution

It introduces a novel, mathematically grounded eDSL for dataflow programming that enhances error detection and scalability compared to current industry tools.

Findings

CircuitFlow outperforms Luigi in scalability with increasing inputs.

It provides static type safety to prevent runtime errors.

The design exemplifies creating modular eDSLs with effects and complex type information.

Abstract

Dataflow applications, such as machine learning algorithms, can run for days, making it desirable to have assurances that they will work correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable run-time errors. This paper presents a new declarative Haskell Embedded DSL (eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder (SMP) on data that models dependencies in the workflow, it has a strong mathematical basis, refocusing on how data flows through an application, resulting in a more expressive solution that not only catches errors statically, but also achieves competitive run-time performance. In our preliminary evaluation, CircuitFlow outperforms the industry-leading Luigi library of Spotify by scaling better with the number of inputs. The innovative creation of CircuitFlow is also of note, exemplifying how to create a modular eDSL whose semantics necessitates effects, and where storing complex type information for program correctness is paramount.