Fast and General Automatic Differentiation for Finite-State Methods

TL;DR

The paper introduces the 'morphism-trick', a novel method for efficient, semiring-agnostic automatic differentiation in finite-state methods, significantly speeding up gradient computations for automata.

Contribution

It presents the 'morphism-trick' for integrating custom vector-Jacobian products, enabling fast, general automatic differentiation for finite-state algorithms.

Findings

Achieves orders of magnitude faster gradient computation for finite automata.

Provides an open-source library implementing the proposed method.

Demonstrates minimal user effort for efficient differentiation.

Abstract

We propose a new method, that we coined the ``morphism-trick'', to integrate custom implementations of vector-Jacobian products in automatic differentiation softwares, applicable to a wide range of semiring-based computations. Our approach leads to efficient and semiring-agnostic implementations of the backward pass of dynamic programming algorithms. For the particular case of finite-state methods, we introduce an algorithm that computes and differentiates the $\oplus$-sum of all paths' weight of a finite-state automaton. Results show that, with minimal effort from the user, our novel library allows computing the gradient of a function w.r.t. to the weights of a finite state automaton orders of magnitude faster than state-of-the-art automatic differentiation systems. Implementations are made available via an open-source library distributed under a permissive license.