Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks

TL;DR

Cavs introduces a vertex-centric programming interface for dynamic neural networks, enabling efficient execution by avoiding repeated graph construction and leveraging static optimizations, resulting in significant speedups over existing frameworks.

Contribution

The paper presents Cavs, a novel system that efficiently supports dynamic neural network models through a vertex-centric interface and optimized execution strategies.

Findings

Cavs achieves nearly ten times faster training speeds than TensorFlow Fold and DyNet.

The system effectively handles dynamic network structures like sequences, trees, and graphs.

Batching and memory management strategies significantly improve performance.

Abstract

Recent deep learning (DL) models have moved beyond static network architectures to dynamic ones, handling data where the network structure changes every example, such as sequences of variable lengths, trees, and graphs. Existing dataflow-based programming models for DL---both static and dynamic declaration---either cannot readily express these dynamic models, or are inefficient due to repeated dataflow graph construction and processing, and difficulties in batched execution. We present Cavs, a vertex-centric programming interface and optimized system implementation for dynamic DL models. Cavs represents dynamic network structure as a static vertex function $\mathcal{F}$ and a dynamic instance-specific graph $\mathcal{G}$, and performs backpropagation by scheduling the execution of $\mathcal{F}$ following the dependencies in $\mathcal{G}$. Cavs bypasses expensive graph construction and preprocessing overhead, allows for the use of static graph optimization techniques on pre-defined operations in $\mathcal{F}$, and naturally exposes batched execution opportunities over different graphs. Experiments comparing Cavs to two state-of-the-art frameworks for dynamic NNs (TensorFlow Fold and DyNet) demonstrate the efficacy of this approach: Cavs achieves a near one order of magnitude speedup on training of various dynamic NN architectures, and ablations demonstrate the contribution of our proposed batching and memory management strategies.