Adjoint Differentiation for generic matrix functions

TL;DR

This paper develops a general formula for the adjoint of matrix functions, enabling efficient differentiation in applications like spectral decomposition, correlation matrices, and regularized linear regression.

Contribution

It introduces a unified adjoint differentiation formula for matrix functions, including specific cases like spectral decomposition and correlation matrices, simplifying complex derivative computations.

Findings

Derived a general adjoint formula for matrix functions

Provided closed-form expressions for spectral decomposition and correlation matrices

Simplified adjoint computations for regularized linear regression

Abstract

We derive a formula for the adjoint $\overline{A}$ of a square-matrix operation of the form $C=f(A)$, where $f$ is holomorphic in the neighborhood of each eigenvalue. We then apply the formula to derive closed-form expressions in particular cases of interest such as the case when we have a spectral decomposition $A=UDU^{-1}$, the spectrum cut-off $C=A_+$ and the Nearest Correlation Matrix routine. Finally, we explain how to simplify the computation of adjoints for regularized linear regression coefficients.