Suki: Choreographed Distributed Dataflow in Rust

TL;DR

Suki is a Rust-based embedded DSL for distributed dataflow programming that offers explicit control over computation placement, enabling optimized, type-safe streaming dataflows with zero-overhead local compute units.

Contribution

It introduces a choreographed programming model in Rust that combines staged programming and traits for explicit placement and type safety in distributed dataflow systems.

Findings

Enables explicit placement of compute in dataflow programs.

Provides a high-level Rust API with zero-overhead compilation.

Supports mapping dataflow to cloud resources safely.

Abstract

Programming models for distributed dataflow have long focused on analytical workloads that allow the runtime to dynamically place and schedule compute logic. Meanwhile, models that enable fine-grained control over placement, such as actors, make global optimization difficult. In this extended abstract, we present Suki, an embedded Rust DSL that lets developers implement streaming dataflow with explicit placement of computation. Key to this choreographic programming approach is our use of staged programming, which lets us expose a high-level Rust API while compiling local compute units into individual binaries with zero-overhead. We also explore how this approach, combined with Rust's trait system, enables a type-safe API for mapping dataflow programs to cloud computing resources.