Improving the Expressiveness of Deep Learning Frameworks with Recursion

TL;DR

This paper introduces recursion into deep learning frameworks, enabling more efficient and natural execution of recursive neural networks by exploiting their hierarchical structure for improved performance.

Contribution

It adds recursive execution capabilities and APIs to existing frameworks like TensorFlow, enhancing their ability to represent and efficiently run recursive neural networks.

Findings

Recursive implementation outperforms iterative methods in training and inference times.

The approach better captures the recursive structure of neural networks.

Resource utilization is improved with recursive execution.

Abstract

Recursive neural networks have widely been used by researchers to handle applications with recursively or hierarchically structured data. However, embedded control flow deep learning frameworks such as TensorFlow, Theano, Caffe2, and MXNet fail to efficiently represent and execute such neural networks, due to lack of support for recursion. In this paper, we add recursion to the programming model of existing frameworks by complementing their design with recursive execution of dataflow graphs as well as additional APIs for recursive definitions. Unlike iterative implementations, which can only understand the topological index of each node in recursive data structures, our recursive implementation is able to exploit the recursive relationships between nodes for efficient execution based on parallel computation. We present an implementation on TensorFlow and evaluation results with various recursive neural network models, showing that our recursive implementation not only conveys the recursive nature of recursive neural networks better than other implementations, but also uses given resources more effectively to reduce training and inference time.