On Study of Mutual Information and its Estimation Methods

TL;DR

This paper provides an extensive review of mutual information, its properties, and the current challenges in estimating it, highlighting its growing importance in deep learning research.

Contribution

It offers a comprehensive analysis of mutual information definitions, properties, and reviews the limitations of existing estimation methods in the context of deep learning.

Findings

Mutual information is increasingly used in deep learning research.

Current estimation methods face significant drawbacks.

Understanding of mutual information properties is crucial for robust model development.

Abstract

The presence of mutual information in the research of deep learning has grown significantly. It has been proven that mutual information can be a good objective function to build a robust deep learning model. Most of the researches utilize estimation methods to approximate the true mutual information. This technical report delivers an extensive study about definitions as well as properties of mutual information. This article then delivers some reviews and current drawbacks of mutual information estimation methods afterward.