Differentiating Through Linear Solvers

TL;DR

This paper empirically investigates the effects of differentiating through linear solvers, challenging the common advice to avoid such differentiation and comparing the accuracy of different approaches.

Contribution

It provides the first systematic empirical comparison of differentiating through linear solvers versus high-level derivative expressions.

Findings

Differentiating through linear solvers can yield accurate results in certain scenarios.

High-level approaches generally maintain better numerical stability.

Empirical results highlight trade-offs between accuracy and computational complexity.

Abstract

Computer programs containing calls to linear solvers are a known challenge for automatic differentiation. Previous publications advise against differentiating through the low-level solver implementation, and instead advocate for high-level approaches that express the derivative in terms of a modified linear system that can be solved with a separate solver call. Despite this ubiquitous advice, we are not aware of prior work comparing the accuracy of both approaches. With this article we thus empirically study a simple question: What happens if we ignore common wisdom, and differentiate through linear solvers?