DataX: A system for Data eXchange and transformation of streams

TL;DR

DataX is a comprehensive platform that simplifies the development and operation of distributed data stream processing applications, leveraging abstractions, automatic communication setup, and serverless computing to enhance productivity and scalability.

Contribution

It introduces DataX, a novel platform that streamlines data stream exchange, transformation, and fusion, reducing programming complexity and enabling scalable, reliable distributed applications.

Findings

Improves programmer productivity in stream processing

Enables easy reuse of microservices and data streams

Supports auto-scaling and reliable operation at scale

Abstract

The exponential growth in smart sensors and rapid progress in 5G networks is creating a world awash with data streams. However, a key barrier to building performant multi-sensor, distributed stream processing applications is high programming complexity. We propose DataX, a novel platform that improves programmer productivity by enabling easy exchange, transformations, and fusion of data streams. DataX abstraction simplifies the application's specification and exposes parallelism and dependencies among the application functions (microservices). DataX runtime automatically sets up appropriate data communication mechanisms, enables effortless reuse of microservices and data streams across applications, and leverages serverless computing to transform, fuse, and auto-scale microservices. DataX makes it easy to write, deploy and reliably operate distributed applications at scale. Synthesizing these capabilities into a single platform is substantially more transformative than any available stream processing system.