Moreau-Yoshida Variational Transport: A General Framework For Solving Regularized Distributional Optimization Problems

TL;DR

This paper introduces Moreau-Yoshida Variational Transport (MYVT), a novel method for solving regularized distributional optimization problems in machine learning, using smooth approximations and saddle point reformulation for efficient computation.

Contribution

The paper proposes MYVT, a new approach combining Moreau-Yoshida envelopes and variational representation to efficiently solve nonsmooth regularized distributional optimization problems.

Findings

The method effectively solves complex regularized problems in experiments.

Theoretical analysis confirms convergence and stability.

Experimental results demonstrate improved performance over existing methods.

Abstract

We consider a general optimization problem of minimizing a composite objective functional defined over a class of probability distributions. The objective is composed of two functionals: one is assumed to possess the variational representation and the other is expressed in terms of the expectation operator of a possibly nonsmooth convex regularizer function. Such a regularized distributional optimization problem widely appears in machine learning and statistics, such as proximal Monte-Carlo sampling, Bayesian inference and generative modeling, for regularized estimation and generation. We propose a novel method, dubbed as Moreau-Yoshida Variational Transport (MYVT), for solving the regularized distributional optimization problem. First, as the name suggests, our method employs the Moreau-Yoshida envelope for a smooth approximation of the nonsmooth function in the objective. Second, we reformulate the approximate problem as a concave-convex saddle point problem by leveraging the variational representation, and then develope an efficient primal-dual algorithm to approximate the saddle point. Furthermore, we provide theoretical analyses and report experimental results to demonstrate the effectiveness of the proposed method.