Differentiate Everything with a Reversible Embeded Domain-Specific Language

TL;DR

This paper introduces a reversible embedded domain-specific language (eDSL) for automatic differentiation that eliminates stack overhead, enabling more efficient back-propagation in industrial applications, especially on GPUs.

Contribution

It proposes a reversible programming eDSL that allows bi-directional execution without stack operations, improving AD efficiency and compatibility with existing compiler optimizations.

Findings

Achieves state-of-the-art performance in sparse matrix AD

Reduces space overhead compared to checkpointing methods

Compatible with GPU kernel compilation and existing compiler features

Abstract

Reverse-mode automatic differentiation (AD) suffers from the issue of having too much space overhead to trace back intermediate computational states for back-propagation. The traditional method to trace back states is called checkpointing that stores intermediate states into a global stack and restore state through either stack pop or re-computing. The overhead of stack manipulations and re-computing makes the general purposed (not tensor-based) AD engines unable to meet many industrial needs. Instead of checkpointing, we propose to use reverse computing to trace back states by designing and implementing a reversible programming eDSL, where a program can be executed bi-directionally without implicit stack operations. The absence of implicit stack operations makes the program compatible with existing compiler features, including utilizing existing optimization passes and compiling the code as GPU kernels. We implement AD for sparse matrix operations and some machine learning applications to show that our framework has the state-of-the-art performance.