Moreau-Yosida $f$-divergences
D\'avid Terj\'ek

TL;DR
This paper introduces a novel framework for approximating and representing $f$-divergences using Moreau-Yosida regularization, enabling flexible variational formulas and practical algorithms for GAN training.
Contribution
It defines the Moreau-Yosida $f$-divergences, generalizes variational formulas, relaxes Lipschitz constraints, and proposes an algorithm for computing the convex conjugate suitable for automatic differentiation.
Findings
Proposes the Moreau-Yosida $f$-GAN with competitive results on CIFAR-10.
Provides a practical algorithm compatible with automatic differentiation.
Offers theoretical insights into the variational representation and Lipschitz function spaces.
Abstract
Variational representations of -divergences are central to many machine learning algorithms, with Lipschitz constrained variants recently gaining attention. Inspired by this, we define the Moreau-Yosida approximation of -divergences with respect to the Wasserstein- metric. The corresponding variational formulas provide a generalization of a number of recent results, novel special cases of interest and a relaxation of the hard Lipschitz constraint. Additionally, we prove that the so-called tight variational representation of -divergences can be to be taken over the quotient space of Lipschitz functions, and give a characterization of functions achieving the supremum in the variational representation. On the practical side, we propose an algorithm to calculate the tight convex conjugate of -divergences compatible with automatic differentiation frameworks. As an application…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsSparse and Compressive Sensing Techniques · Adversarial Robustness in Machine Learning · Statistical Mechanics and Entropy
