Differentiable Programming: Efficient Smoothing of Control-Flow-Induced Discontinuities

TL;DR

This paper introduces a method to make programs with control flow discontinuities differentiable by interpolating trajectories at control points, enabling more accurate derivatives in discontinuous code.

Contribution

It presents a novel, efficient approach to interpolate control-flow boundaries, allowing automatic and smooth differentiation of programs with discontinuities.

Findings

Method effectively interpolates control-flow boundaries

Enables accurate derivatives in discontinuous programs

Applicable to complex nested control structures

Abstract

We want to obtain derivatives in discontinuous program code, where default Algorithmic Differentiation may not perform well. Specifically, we consider discontinuities induced by control flow statements, where meaningful derivatives should ideally be capable of representing the resulting kinks in the trajectory. To achieve this, one can interpolate the trajectory at the control flow statements before taking the derivative. We formulate a method to efficiently interpolate between all boundaries induced by control flow in program code. Theoretically, code can be viewed as a series of piecewise continuous functions applied in succession. These functions are nested inside one another and result in a function composition with several cases. We interpret this function composition as a tree and devise a heuristic to identify paths that are relevant to the interpolation. This allows us to conceive a language that smoothly interpolates control-flow statements automatically and efficiently, making it fully differentiable.