One-step differentiation of iterative algorithms

TL;DR

This paper introduces a one-step differentiation method for iterative algorithms that combines ease of implementation with high performance, reducing computational costs in automatic differentiation.

Contribution

It presents a novel Jacobian-free backpropagation approach that is as simple as automatic differentiation and as efficient as implicit differentiation for fast algorithms.

Findings

Theoretical analysis confirms the accuracy of the one-step estimator.

Numerical experiments demonstrate the method's effectiveness in various algorithms.

Application to bilevel optimization shows practical benefits.

Abstract

In appropriate frameworks, automatic differentiation is transparent to the user at the cost of being a significant computational burden when the number of operations is large. For iterative algorithms, implicit differentiation alleviates this issue but requires custom implementation of Jacobian evaluation. In this paper, we study one-step differentiation, also known as Jacobian-free backpropagation, a method as easy as automatic differentiation and as performant as implicit differentiation for fast algorithms (e.g., superlinear optimization methods). We provide a complete theoretical approximation analysis with specific examples (Newton's method, gradient descent) along with its consequences in bilevel optimization. Several numerical examples illustrate the well-foundness of the one-step estimator.