A Reverse Jensen Inequality Result with Application to Mutual Information Estimation

TL;DR

This paper introduces a reversed form of Jensen's inequality under minimal constraints, enabling a new variational approach to mutual information estimation with improved training performance.

Contribution

It presents a novel reverse Jensen inequality applicable in probabilistic settings, leading to a new mutual information estimator with better training behavior.

Findings

Reversed Jensen inequality under minimal constraints.

New variational mutual information estimator.

Improved training performance over existing estimators.

Abstract

The Jensen inequality is a widely used tool in a multitude of fields, such as for example information theory and machine learning. It can be also used to derive other standard inequalities such as the inequality of arithmetic and geometric means or the H\"older inequality. In a probabilistic setting, the Jensen inequality describes the relationship between a convex function and the expected value. In this work, we want to look at the probabilistic setting from the reverse direction of the inequality. We show that under minimal constraints and with a proper scaling, the Jensen inequality can be reversed. We believe that the resulting tool can be helpful for many applications and provide a variational estimation of mutual information, where the reverse inequality leads to a new estimator with superior training behavior compared to current estimators.