Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs

TL;DR

This paper provides a theoretical framework for analyzing the complexity of feature transformations in ReLU DNNs, revealing their relation to disentanglement and implications for training and robustness.

Contribution

It introduces new information-theoretic metrics for complexity, analyzes their behavior during training, and explores their impact on overfitting and adversarial robustness.

Findings

Complexity correlates strongly with transformation disentanglement.

Training dynamics show characteristic changes in complexity.

Complexity metrics can guide DNN training to improve robustness.

Abstract

This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN's complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN.