Understanding Automatic Differentiation Pitfalls

TL;DR

This paper categorizes common pitfalls in automatic differentiation, illustrating how incorrect derivatives can arise in various scenarios, and reviews debugging techniques to improve reliability and understanding of AD tools.

Contribution

It provides a comprehensive categorization of AD pitfalls with practical examples and evaluates debugging methods to help users avoid and detect errors.

Findings

Pitfalls include chaos, oscillations, discretizations, fixed-point loops, lookup tables, and linear solvers.

Debugging techniques vary in effectiveness across different problematic cases.

Awareness of pitfalls improves the reliability of automatic differentiation applications.

Abstract

Automatic differentiation, also known as backpropagation, AD, autodiff, or algorithmic differentiation, is a popular technique for computing derivatives of computer programs accurately and efficiently. Sometimes, however, the derivatives computed by AD could be interpreted as incorrect. These pitfalls occur systematically across tools and approaches. In this paper we broadly categorize problematic usages of AD and illustrate each category with examples such as chaos, time-averaged oscillations, discretizations, fixed-point loops, lookup tables, and linear solvers. We also review debugging techniques and their effectiveness in these situations. With this article we hope to help readers avoid unexpected behavior, detect problems more easily when they occur, and have more realistic expectations from AD tools.