Gradients are Not All You Need

TL;DR

This paper discusses a failure mode in differentiable programming related to Jacobian spectrum issues, affecting various applications like RNNs and physics simulations, and provides criteria to predict such failures.

Contribution

It identifies a chaos-based failure mode linked to Jacobian spectra in differentiable systems and offers practical criteria to anticipate these issues.

Findings

Jacobian spectrum influences differentiable system stability

Failure modes occur in RNNs, physics simulations, and learned optimizers

Criteria can predict when differentiation will fail due to chaos

Abstract

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.