Derivative of the truncated singular value and eigen decomposition

TL;DR

This paper discusses the derivation of derivatives for truncated singular value and eigenvalue decompositions, crucial for stable gradient computations in machine learning and physics applications.

Contribution

It provides a detailed derivation of derivatives for truncated decompositions, addressing challenges when the full decomposition is unknown.

Findings

Clarifies how to compute derivatives using only truncated parts

Builds on previous work with comprehensive derivations

Enhances stability in gradient computations for applications

Abstract

Recently developed applications in the field of machine learning and computational physics rely on automatic differentiation techniques, that require stable and efficient linear algebra gradient computations. This technical note provides a comprehensive and detailed discussion of the derivative of the truncated singular and eigenvalue decomposition. It summarizes previous work and builds on them with an extensive description of how to derive the relevant terms. A main focus is correctly expressing the derivative in terms of the truncated part, despite lacking knowledge of the full decomposition.