Differentiable Programming \`a la Moreau

TL;DR

This paper develops a differentiable programming framework based on the Moreau envelope, enabling advanced analysis and integration of optimization techniques within deep learning systems.

Contribution

It introduces a compositional calculus for Moreau envelopes and demonstrates how to incorporate it into differentiable programming for machine learning.

Findings

Framework enables integration of Moreau envelopes into deep networks.

Mathematical optimization perspective on gradient back-propagation variants.

Facilitates analysis of optimization algorithms in deep learning.

Abstract

The notion of a Moreau envelope is central to the analysis of first-order optimization algorithms for machine learning. Yet, it has not been developed and extended to be applied to a deep network and, more broadly, to a machine learning system with a differentiable programming implementation. We define a compositional calculus adapted to Moreau envelopes and show how to integrate it within differentiable programming. The proposed framework casts in a mathematical optimization framework several variants of gradient back-propagation related to the idea of the propagation of virtual targets.