Fenchel-Young Variational Learning

TL;DR

This paper introduces Fenchel-Young variational learning, a new framework that generalizes classical variational methods using FY divergences, enabling more flexible models with novel features like adaptive sparsity.

Contribution

The paper develops a general class of FY variational methods, deriving algorithms and extending classical models with new divergence-based formulations and sparsity features.

Findings

FY variational methods outperform classical counterparts in experiments.

FYEM exhibits adaptive sparsity in the E-step.

FYVAE supports sparse observations and posteriors.

Abstract

From a variational perspective, many statistical learning criteria involve seeking a distribution that balances empirical risk and regularization. In this paper, we broaden this perspective by introducing a new general class of variational methods based on Fenchel-Young (FY) losses, treated as divergences that generalize (and encompass) the familiar Kullback-Leibler divergence at the core of classical variational learning. Our proposed formulation -- FY variational learning -- includes as key ingredients new notions of FY free energy, FY evidence, FY evidence lower bound, and FY posterior. We derive alternating minimization and gradient backpropagation algorithms to compute (or lower bound) the FY evidence, which enables learning a wider class of models than previous variational formulations. This leads to generalized FY variants of classical algorithms, such as an FY expectation-maximization (FYEM) algorithm, and latent-variable models, such as an FY variational autoencoder (FYVAE). Our new methods are shown to be empirically competitive, often outperforming their classical counterparts, and most importantly, to have qualitatively novel features. For example, FYEM has an adaptively sparse E-step, while the FYVAE can support models with sparse observations and sparse posteriors.