Efficient Differentiable Programming in a Functional Array-Processing Language

TL;DR

This paper introduces a system for automatic differentiation in a functional array-processing language that combines source-to-source differentiation with global optimizations, leading to improved performance on machine learning and computer vision benchmarks.

Contribution

The paper presents a novel system that integrates differentiable programming with global loop optimizations in a higher-order functional language, enhancing efficiency.

Findings

Outperforms state-of-the-art AD tools on real-world benchmarks

Supports both source-to-source differentiation and loop transformations

Demonstrates efficiency gains in machine learning and computer vision tasks

Abstract

We present a system for the automatic differentiation of a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source automatic differentiation and global optimizations such as loop transformations. Thanks to this feature, we demonstrate how for some real-world machine learning and computer vision benchmarks, the system outperforms the state-of-the-art automatic differentiation tools.