Quantitative Performance Assessment of CNN Units via Topological Entropy Calculation

TL;DR

This paper introduces a topological entropy measure to quantitatively assess the status of individual CNN units, providing insights into their role and the network's generalization ability.

Contribution

A novel algebraic topological method using feature entropy to evaluate CNN unit status across different models and training stages.

Findings

Feature entropy decreases with network depth.

Feature entropy correlates with training loss.

Higher feature entropy indicates better feature representation.

Abstract

Identifying the status of individual network units is critical for understanding the mechanism of convolutional neural networks (CNNs). However, it is still challenging to reliably give a general indication of unit status, especially for units in different network models. To this end, we propose a novel method for quantitatively clarifying the status of single unit in CNN using algebraic topological tools. Unit status is indicated via the calculation of a defined topological-based entropy, called feature entropy, which measures the degree of chaos of the global spatial pattern hidden in the unit for a category. In this way, feature entropy could provide an accurate indication of status for units in different networks with diverse situations like weight-rescaling operation. Further, we show that feature entropy decreases as the layer goes deeper and shares almost simultaneous trend with loss during training. We show that by investigating the feature entropy of units on only training data, it could give discrimination between networks with different generalization ability from the view of the effectiveness of feature representations.