Coarsening Optimization for Differentiable Programming

TL;DR

This paper introduces coarsening optimization, a new method that combines symbolic and algorithmic differentiation to significantly speed up differentiable programming by reducing computation granularity and managing control flow complexities.

Contribution

It proposes a systematic coarsening approach for AD, introduces phi-calculus for handling control flow, and demonstrates substantial speedups in real-world applications.

Findings

Achieves up to two orders of magnitude speedup in AD

Effectively manages control flow in symbolic differentiation

Reduces runtime computations and data allocations

Abstract

This paper presents a novel optimization for differentiable programming named coarsening optimization. It offers a systematic way to synergize symbolic differentiation and algorithmic differentiation (AD). Through it, the granularity of the computations differentiated by each step in AD can become much larger than a single operation, and hence lead to much reduced runtime computations and data allocations in AD. To circumvent the difficulties that control flow creates to symbolic differentiation in coarsening, this work introduces phi-calculus, a novel method to allow symbolic reasoning and differentiation of computations that involve branches and loops. It further avoids "expression swell" in symbolic differentiation and balance reuse and coarsening through the design of reuse-centric segment of interest identification. Experiments on a collection of real-world applications show that coarsening optimization is effective in speeding up AD, producing several times to two orders of magnitude speedups.