Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimation

TL;DR

This paper introduces a systematic method to create tighter variational representations of divergences, improving the efficiency and accuracy of statistical estimation in high-dimensional data using neural networks.

Contribution

It develops a new approach for constructing tighter variational divergence representations using auxiliary optimization and curvature analysis, enhancing learning speed and estimation accuracy.

Findings

Tighter variational representations lead to significantly faster divergence estimation.

The methodology improves estimation accuracy in high-dimensional datasets.

Neural network optimization demonstrates nearly an order of magnitude acceleration.

Abstract

Variational representations of divergences and distances between high-dimensional probability distributions offer significant theoretical insights and practical advantages in numerous research areas. Recently, they have gained popularity in machine learning as a tractable and scalable approach for training probabilistic models and for statistically differentiating between data distributions. Their advantages include: 1) They can be estimated from data as statistical averages. 2) Such representations can leverage the ability of neural networks to efficiently approximate optimal solutions in function spaces. However, a systematic and practical approach to improving the tightness of such variational formulas, and accordingly accelerate statistical learning and estimation from data, is currently lacking. Here we develop such a methodology for building new, tighter variational representations of divergences. Our approach relies on improved objective functionals constructed via an auxiliary optimization problem. Furthermore, the calculation of the functional Hessian of objective functionals unveils the local curvature differences around the common optimal variational solution; this quantifies and orders the tightness gains between different variational representations. Finally, numerical simulations utilizing neural network optimization demonstrate that tighter representations can result in significantly faster learning and more accurate estimation of divergences in both synthetic and real datasets (of more than 1000 dimensions), often accelerated by nearly an order of magnitude.