Learning to Optimize by Differentiable Programming

TL;DR

This paper explores how differentiable programming frameworks can be used to learn and improve optimization algorithms, enhancing convergence and solution quality across various applications.

Contribution

It introduces a paradigm shift where optimization algorithms are learned within differentiable programming frameworks, guided by duality principles.

Findings

Learned duality-informed schemes improve convergence.

End-to-end training enhances solution quality.

Case studies demonstrate practical benefits.

Abstract

Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as ADMM and PDHG can be learned and adapted. Case studies across LP, OPF, Laplacian regularization, and neural network verification illustrate these gains.