Least-Squares Problem Over Probability Measure Space

TL;DR

This paper explores a variational problem over probability measures, analyzing how different divergence choices affect the recovery of conditional or marginal distributions in a probabilistic setting.

Contribution

It introduces a framework for least-squares problems over probability measure spaces and characterizes the effects of different divergence measures on the solutions.

Findings

Using $\, ext{phi}$-divergence recovers the conditional distribution.

Using Wasserstein distance recovers the marginal distribution.

The choice of divergence influences the type of distribution recovered.

Abstract

In this work, we investigate the variational problem $$\rho_x^\ast = \text{argmin}_{\rho_x} D(G\#\rho_x, \rho_y)\,, $$ where $D$ quantifies the difference between two probability measures, and ${G}$ is a forward operator that maps a variable $x$ to $y=G(x)$. This problem can be regarded as an analogue of its counterpart in linear spaces (e.g., Euclidean spaces), $\text{argmin}_x \|G(x) - y\|^2$. Similar to how the choice of norm $\|\cdot\|$ influences the optimizer in $\mathbb R^d$ or other linear spaces, the minimizer in the probabilistic variational problem also depends on the choice of $D$. Our findings reveal that using a $\phi$-divergence for $D$ leads to the recovery of a conditional distribution of $\rho_y$, while employing the Wasserstein distance results in the recovery of a marginal distribution.