A Brief Introduction to Automatic Differentiation for Machine Learning

TL;DR

This paper introduces automatic differentiation, a key technique in machine learning frameworks like TensorFlow and PyTorch, explaining its motivations, implementations, and role in enabling efficient gradient-based optimization for neural networks.

Contribution

It provides a comprehensive overview of automatic differentiation, including its motivations, various implementation approaches, and practical examples using popular frameworks.

Findings

Automatic differentiation simplifies derivative calculations in neural network training.

Different implementation approaches of AD are discussed and compared.

Practical examples demonstrate AD's application in TensorFlow and PyTorch.

Abstract

Machine learning and neural network models in particular have been improving the state of the art performance on many artificial intelligence related tasks. Neural network models are typically implemented using frameworks that perform gradient based optimization methods to fit a model to a dataset. These frameworks use a technique of calculating derivatives called automatic differentiation (AD) which removes the burden of performing derivative calculations from the model designer. In this report we describe AD, its motivations, and different implementation approaches. We briefly describe dataflow programming as it relates to AD. Lastly, we present example programs that are implemented with Tensorflow and PyTorch, which are two commonly used AD frameworks.