Understanding Neural Networks with Logarithm Determinant Entropy Estimator

TL;DR

This paper introduces the LogDet estimator, a reliable matrix-based entropy measure, to analyze neural network behavior and distinguish functional differences between shallow and deep layers, supporting the information bottleneck theory.

Contribution

The paper proposes the LogDet estimator as a new, reliable entropy estimator for neural networks, addressing issues with previous methods and enabling better analysis of network information processing.

Findings

LogDet estimator effectively estimates entropy in diverse neural network distributions.

It reveals functional distinctions between shallow and deep layers.

Supports the compression phenomenon in the information bottleneck theory.

Abstract

Understanding the informative behaviour of deep neural networks is challenged by misused estimators and the complexity of network structure, which leads to inconsistent observations and diversified interpretation. Here we propose the LogDet estimator -- a reliable matrix-based entropy estimator that approximates Shannon differential entropy. We construct informative measurements based on LogDet estimator, verify our method with comparable experiments and utilize it to analyse neural network behaviour. Our results demonstrate the LogDet estimator overcomes the drawbacks that emerge from highly diverse and degenerated distribution thus is reliable to estimate entropy in neural networks. The Network analysis results also find a functional distinction between shallow and deeper layers, which can help understand the compression phenomenon in the Information bottleneck theory of neural networks.