Self-similarity Analysis in Deep Neural Networks

TL;DR

This paper investigates the self-similarity in deep neural networks' feature spaces, proposing a complex network model to analyze its impact on classification performance across various architectures, and demonstrates that controlling self-similarity can enhance accuracy.

Contribution

It introduces a novel complex network modeling approach to quantify and manipulate self-similarity in hidden layers, improving deep neural network performance.

Findings

Self-similarity varies across network architectures.

Embedding self-similarity constraints improves accuracy by up to 6%.

Analysis applies to MLP, convolutional, and attention networks.

Abstract

Current research has found that some deep neural networks exhibit strong hierarchical self-similarity in feature representation or parameter distribution. However, aside from preliminary studies on how the power-law distribution of weights across different training stages affects model performance,there has been no quantitative analysis on how the self-similarity of hidden space geometry influences model weight optimization, nor is there a clear understanding of the dynamic behavior of internal neurons. Therefore, this paper proposes a complex network modeling method based on the output features of hidden-layer neurons to investigate the self-similarity of feature networks constructed at different hidden layers, and analyzes how adjusting the degree of self-similarity in feature networks can enhance the classification performance of deep neural networks. Validated on three types of networks MLP architectures, convolutional networks, and attention architectures this study reveals that the degree of self-similarity exhibited by feature networks varies across different model architectures. Furthermore, embedding constraints on the self-similarity of feature networks during the training process can improve the performance of self-similar deep neural networks (MLP architectures and attention architectures) by up to 6 percentage points.