Understanding the Dynamics of DNNs Using Graph Modularity

TL;DR

This paper models the evolution of feature representations in deep neural networks as dynamic graphs and uses modularity to analyze layer-wise community structures, providing insights into redundancy and pruning strategies.

Contribution

It introduces a graph-theoretic approach using modularity to understand DNN layer dynamics and offers a theoretical basis for layer pruning based on community evolution.

Findings

Modularity tends to increase with depth in DNNs.

Degradation and plateau in modularity indicate redundant layers.

Pruning based on modularity has minimal impact on performance.

Abstract

There are good arguments to support the claim that deep neural networks (DNNs) capture better feature representations than the previous hand-crafted feature engineering, which leads to a significant performance improvement. In this paper, we move a tiny step towards understanding the dynamics of feature representations over layers. Specifically, we model the process of class separation of intermediate representations in pre-trained DNNs as the evolution of communities in dynamic graphs. Then, we introduce modularity, a generic metric in graph theory, to quantify the evolution of communities. In the preliminary experiment, we find that modularity roughly tends to increase as the layer goes deeper and the degradation and plateau arise when the model complexity is great relative to the dataset. Through an asymptotic analysis, we prove that modularity can be broadly used for different applications. For example, modularity provides new insights to quantify the difference between feature representations. More crucially, we demonstrate that the degradation and plateau in modularity curves represent redundant layers in DNNs and can be pruned with minimal impact on performance, which provides theoretical guidance for layer pruning. Our code is available at https://github.com/yaolu-zjut/Dynamic-Graphs-Construction.