Causify DataFlow: A Framework For High-performance Machine Learning Stream Computing

TL;DR

DataFlow is a unified framework that enables high-performance, reproducible, and causality-preserving machine learning on streaming time-series data, bridging the gap between batch and real-time systems.

Contribution

It introduces a DAG-based execution model with point-in-time idempotency, ensuring consistent behavior across batch and streaming modes without code changes.

Findings

Ensures identical batch and streaming execution without code modifications.

Supports flexible tiling for different temporal and feature granularities.

Demonstrates effectiveness in finance, IoT, fraud detection, and analytics domains.

Abstract

We present DataFlow, a computational framework for building, testing, and deploying high-performance machine learning systems on unbounded time-series data. Traditional data science workflows assume finite datasets and require substantial reimplementation when moving from batch prototypes to streaming production systems. This gap introduces causality violations, batch boundary artifacts, and poor reproducibility of real-time failures. DataFlow resolves these issues through a unified execution model based on directed acyclic graphs (DAGs) with point-in-time idempotency: outputs at any time t depend only on a fixed-length context window preceding t. This guarantee ensures that models developed in batch mode execute identically in streaming production without code changes. The framework enforces strict causality by automatically tracking knowledge time across all transformations, eliminating future-peeking bugs. DataFlow supports flexible tiling across temporal and feature dimensions, allowing the same model to operate at different frequencies and memory profiles via configuration alone. It integrates natively with the Python data science stack and provides fit/predict semantics for online learning, caching and incremental computation, and automatic parallelization through DAG-based scheduling. We demonstrate its effectiveness across domains including financial trading, IoT, fraud detection, and real-time analytics.