Automatic Differentiation: Inverse Accumulation Mode

TL;DR

This paper introduces a method for efficiently computing inverse Jacobian-vector products in automatic differentiation, leveraging sparsity invariance to enable faster Newton step calculations under certain conditions.

Contribution

It presents a novel approach to compute inverse Jacobian products efficiently by exploiting the invariance of sparsity patterns, expanding the capabilities of automatic differentiation.

Findings

Efficient inverse Jacobian-vector product computation is possible under certain conditions.

The method leverages sparsity invariance of Jacobian matrices.

Potential applications include faster Newton steps and numerical calculations.

Abstract

We show that, under certain circumstances, it is possible to automatically compute Jacobian-inverse-vector and Jacobian-inverse-transpose-vector products about as efficiently as Jacobian-vector and Jacobian-transpose-vector products. The key insight is to notice that the Jacobian corresponding to the use of one basis function is of a form whose sparsity is invariant to inversion. The main restriction of the method is a constraint on the number of active variables, which suggests a variety of techniques or generalization to allow the constraint to be enforced or relaxed. This technique has the potential to allow the efficient direct calculation of Newton steps as well as other numeric calculations of interest.