Software-based Automatic Differentiation is Flawed

TL;DR

This paper argues that current software implementations of automatic differentiation, which rely on object-oriented programming and lack expression simplification, can produce unbounded errors, questioning their reliability.

Contribution

It highlights a fundamental flaw in existing automatic differentiation frameworks that do not simplify expressions, leading to potentially unbounded errors.

Findings

Unbounded errors can occur in current automatic differentiation software.

Lack of expression simplification is a key issue.

Object-oriented approaches do not inherently prevent errors.

Abstract

Various software efforts embrace the idea that object oriented programming enables a convenient implementation of the chain rule, facilitating so-called automatic differentiation via backpropagation. Such frameworks have no mechanism for simplifying the expressions (obtained via the chain rule) before evaluating them. As we illustrate below, the resulting errors tend to be unbounded.