Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding
Wentao Huang, Kechen Zhang
TL;DR
This paper introduces new approximation formulas for Shannon mutual information applicable to discrete variables, demonstrating high accuracy in neural coding contexts and offering practical tools for information theory applications.
Contribution
The paper develops and validates novel approximation formulas for mutual information that work for both discrete and continuous variables, enhancing computational feasibility in neural coding analysis.
Findings
Approximation formulas are highly accurate for neural population responses.
One formula performs well regardless of variable discreteness.
Numerical simulations confirm practical applicability.
Abstract
Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback--Leibler divergence and R\'{e}nyi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded…
Click any figure to enlarge with its caption.
Figure 1
Figure 1
Figure 1
Figure 1
Figure 2
Figure 2
Figure 2
Figure 2
Figure 3
Figure 3
Figure 3
Figure 3
Figure 4
Figure 4
Figure 4
Figure 4
Figure 5
Figure 5
Figure 5
Figure 5
Figure 6
Figure 6
Figure 6
Figure 6Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.