Automating Variational Differentiation

TL;DR

This paper introduces a novel theoretical model for variational differentiation that enhances analytic backpropagation and can handle complex variational problems in physics and chemistry, overcoming limitations of existing AD methods.

Contribution

It proposes a new differentiation framework based on combinatory logic, enabling more flexible and efficient variational differentiation in high-performance computing environments.

Findings

Developed CombDiff system for variational problems

Successfully applied to Hartree-Fock theory and neural networks

Supports complex numbers and analytic backpropagation

Abstract

Many problems in Physics and Chemistry are formulated as the minimization of a functional. Therefore, methods for solving these problems typically require differentiating maps whose input and/or output are functions -- commonly referred to as variational differentiation. Such maps are not addressed at the mathematical level by the chain rule, which underlies modern symbolic and algorithmic differentiation (AD) systems. Although there are algorithmic solutions such as tracing and reverse accumulation, they do not provide human readability and introduce strict programming constraints that bottleneck performance, especially in high-performance computing (HPC) environments. In this manuscript, we propose a new computer theoretic model of differentiation by combining the pullback of the $\mathbf{B}$ and $\mathbf{C}$ combinators from the combinatory logic. Unlike frameworks based on the chain rule, this model differentiates a minimal complete basis for the space of computable functions. Consequently, the model is capable of analytic backpropagation and variational differentiation while supporting complex numbers. To demonstrate the generality of this approach we build a system named CombDiff, which can differentiate nontrivial variational problems such as Hartree-Fock (HF) theory and multilayer perceptrons.