FastAD: Expression Template-Based C++ Library for Fast and Memory-Efficient Automatic Differentiation

TL;DR

FastAD is a C++ library for automatic differentiation that significantly improves speed and memory efficiency over existing libraries through expression templates and compile-time optimizations.

Contribution

Introduces FastAD, a C++ template library that enhances automatic differentiation performance and memory efficiency using vectorization and expression templates.

Findings

FastAD is 2-10 times faster than Adept.

FastAD is 2-19 times faster than Stan.

FastAD reduces memory consumption compared to existing libraries.

Abstract

Automatic differentiation is a set of techniques to efficiently and accurately compute the derivative of a function represented by a computer program. Existing C++ libraries for automatic differentiation (e.g. Adept, Stan Math Library), however, exhibit large memory consumptions and runtime performance issues. This paper introduces FastAD, a new C++ template library for automatic differentiation, that overcomes all of these challenges in existing libraries by using vectorization, simpler memory management using a fully expression-template-based design, and other compile-time optimizations to remove some run-time overhead. Benchmarks show that FastAD performs 2-10 times faster than Adept and 2-19 times faster than Stan across various test cases including a few real-world examples.