Recursive Function Definitions in Static Dataflow Graphs and their Implementation in TensorFlow

TL;DR

This paper introduces an efficient method for implementing recursive functions in static dataflow graphs like TensorFlow, enabling better support for complex neural network architectures while maintaining efficiency and compatibility with automatic differentiation.

Contribution

It proposes a novel technique that transforms recursive definitions into static graphs using tagging and additional dataflow operations, facilitating distributed execution.

Findings

The technique enables recursive functions in static dataflow graphs.

It maintains compatibility with automatic differentiation.

Experimental results show improved efficiency with tagging.

Abstract

Modern machine learning systems represent their computations as dataflow graphs. The increasingly complex neural network architectures crave for more powerful yet efficient programming abstractions. In this paper we propose an efficient technique for supporting recursive function definitions in dataflow-based systems such as TensorFlow. The proposed approach transforms the given recursive definitions into a static dataflow graph that is enriched with two simple yet powerful dataflow operations. Since static graphs do not change during execution, they can be easily partitioned and executed efficiently in distributed and heterogeneous environments. The proposed technique makes heavy use of the idea of tagging, which was one of the cornerstones of dataflow systems since their inception. We demonstrate that our technique is compatible with the idea of automatic differentiation, a notion that is crucial for dataflow systems that focus on deep learning applications. We describe the principles of an actual implementation of the technique in the TensorFlow framework, and present experimental results that demonstrate that the use of tagging is of paramount importance for developing efficient high-level abstractions for modern dataflow systems.