Neural Capacity Estimators: How Reliable Are They?

TL;DR

This paper rigorously benchmarks neural mutual information estimators like MINE, SMILE, and DINE across different channels to assess their reliability, stability, and effectiveness in capacity estimation tasks.

Contribution

It provides a comprehensive comparison of neural mutual information estimators and offers practical insights into their performance and stability for capacity estimation.

Findings

MINE, SMILE, and DINE vary in stability and accuracy.

Neural estimators can approach channel capacity under certain conditions.

Training stability and initialization sensitivity are critical factors.

Abstract

Recently, several methods have been proposed for estimating the mutual information from sample data using deep neural networks and without the knowing closed form distribution of the data. This class of estimators is referred to as neural mutual information estimators. Although very promising, such techniques have yet to be rigorously bench-marked so as to establish their efficacy, ease of implementation, and stability for capacity estimation which is joint maximization frame-work. In this paper, we compare the different techniques proposed in the literature for estimating capacity and provide a practitioner perspective on their effectiveness. In particular, we study the performance of mutual information neural estimator (MINE), smoothed mutual information lower-bound estimator (SMILE), and directed information neural estimator (DINE) and provide insights on InfoNCE. We evaluated these algorithms in terms of their ability to learn the input distributions that are capacity approaching for the AWGN channel, the optical intensity channel, and peak power-constrained AWGN channel. For both scenarios, we provide insightful comments on various aspects of the training process, such as stability, sensitivity to initialization.