Combinatory Adjoints and Differentiation

TL;DR

This paper introduces a categorical, compositional approach to automatic and symbolic differentiation that uses linear functions and a domain-specific language, enabling efficient adjoint computation without matrices.

Contribution

It develops a novel framework combining combinatory differentiation with adjoint calculus, avoiding matrices and enhancing efficiency for high-dimensional derivatives.

Findings

Symbolic and automatic differentiation are unified through a differential calculus for linear functions.

A domain-specific language effectively represents linear functions in the differentiation process.

The approach is equivalent to reverse-mode automatic differentiation and improves efficiency on high-dimensional spaces.

Abstract

We develop a compositional approach for automatic and symbolic differentiation based on categorical constructions in functional analysis where derivatives are linear functions on abstract vectors rather than being limited to scalars, vectors, matrices or tensors represented as multi-dimensional arrays. We show that both symbolic and automatic differentiation can be performed using a differential calculus for generating linear functions representing Fr\'echet derivatives based on rules for primitive, constant, linear and bilinear functions as well as their sequential and parallel composition. Linear functions are represented in a combinatory domain-specific language. Finally, we provide a calculus for symbolically computing the adjoint of a derivative without using matrices, which are too inefficient to use on high-dimensional spaces. The resulting symbolic representation of a derivative retains the data-parallel operations from the input program. The combination of combinatory differentiation and computing formal adjoints turns out to be behaviorally equivalent to reverse-mode automatic differentiation. In particular, it provides opportunities for optimizations where matrices are too inefficient to represent linear functions.