Dynamic Path Contraction for Distributed, Dynamic Dataflow Languages

TL;DR

This paper introduces dynamic path contraction, a novel runtime algorithm for optimizing distributed, dynamic dataflow programs by applying and reversing optimizations during execution, demonstrated on an actor-based model.

Contribution

It presents a new algorithm for runtime optimization of distributed dataflow programs that adapts dynamically during execution, enhancing performance.

Findings

Preliminary results show potential benefits of dynamic path contraction.

The technique is demonstrated on the Lasp actor-based model.

Runtime tracking enables effective optimization reversibility.

Abstract

We present a work in progress report on applying deforestation to distributed, dynamic dataflow programming models. We propose a novel algorithm, dynamic path contraction, that applies and reverses optimizations to a distributed dataflow application as the program executes. With this algorithm, data and control flow is tracked by the runtime system used to identify potential optimizations as the system is running. We demonstrate and present preliminary results regarding this technique on an actor-based distributed programming model, Lasp, implemented on the Erlang virtual machine.