Differentiation of the Cholesky decomposition

TL;DR

This paper reviews methods for differentiating matrix computations involving the Cholesky decomposition, proposing new algorithms that are faster and more efficient, especially for large matrices, by leveraging block algorithms and combining symbolic and algorithmic approaches.

Contribution

It introduces new cache-friendly, parallelizable algorithms for differentiating Cholesky decompositions, improving speed over existing methods, and combines symbolic and algorithmic strategies for optimal performance.

Findings

New blocked algorithms are faster for large matrices.

Differentiation methods outperform common algorithms by an order of magnitude.

Symbolic approaches are faster for small matrices in some environments.

Abstract

We review strategies for differentiating matrix-based computations, and derive symbolic and algorithmic update rules for differentiating expressions containing the Cholesky decomposition. We recommend new `blocked' algorithms, based on differentiating the Cholesky algorithm DPOTRF in the LAPACK library, which uses `Level 3' matrix-matrix operations from BLAS, and so is cache-friendly and easy to parallelize. For large matrices, the resulting algorithms are the fastest way to compute Cholesky derivatives, and are an order of magnitude faster than the algorithms in common usage. In some computing environments, symbolically-derived updates are faster for small matrices than those based on differentiating Cholesky algorithms. The symbolic and algorithmic approaches can be combined to get the best of both worlds.