Computing Sparse Jacobians and Hessians Using Algorithmic Differentiation

TL;DR

This paper introduces new methods, especially the subgraph method, for efficiently computing large sparse Jacobians and Hessians using algorithmic differentiation, with analysis and comparisons demonstrating potential speedups.

Contribution

The paper presents a novel subgraph method for sparse Jacobian and Hessian computation that eliminates the need for coloring and compression, improving efficiency.

Findings

Subgraph method can be significantly faster for some problems.

Setup time includes graph recording, sparsity, coloring, compression, and optimization.

Experiments show subgraph method has similar or faster run times compared to existing methods.

Abstract

Stochastic scientific models and machine learning optimization estimators have a large number of variables; hence computing large sparse Jacobians and Hessians is important. Algorithmic differentiation (AD) greatly reduces the programming effort required to obtain the sparsity patterns and values for these matrices. We present forward, reverse, and subgraph methods for computing sparse Jacobians and Hessians. Special attention is given the the subgraph method because it is new. The coloring and compression steps are not necessary when computing sparse Jacobians and Hessians using subgraphs. Complexity analysis shows that for some problems the subgraph method is expected to be much faster. We compare C++ operator overloading implementations of the methods in the ADOL-C and CppAD software packages using some of the MINPACK-2 test problems. The experiments are set up in a way that makes them easy to run on different hardware, different systems, different compilers, other test problem and other AD packages. The setup time is the time to record the graph, compute sparsity, coloring, compression, and optimization of the graph. If the setup is necessary for each evaluation, the subgraph implementation has similar run times for sparse Jacobians and faster run times for sparse Hessians.