Correlation Analysis between the Robustness of Sparse Neural Networks and their Random Hidden Structural Priors

TL;DR

This paper investigates how the structural properties of sparse neural networks, modeled as random graphs, relate to their robustness against adversarial attacks, revealing weak correlations with certain graph metrics.

Contribution

It introduces an empirical study linking graph theoretic properties of sparse neural networks to their robustness, highlighting weak but notable correlations.

Findings

Higher graph densities correlate with lower robustness.

Longer average path lengths and node eccentricities negatively correlate with robustness.

Robustness is independent of initialization methods.

Abstract

Deep learning models have been shown to be vulnerable to adversarial attacks. This perception led to analyzing deep learning models not only from the perspective of their performance measures but also their robustness to certain types of adversarial attacks. We take another step forward in relating the architectural structure of neural networks from a graph theoretic perspective to their robustness. We aim to investigate any existing correlations between graph theoretic properties and the robustness of Sparse Neural Networks. Our hypothesis is, that graph theoretic properties as a prior of neural network structures are related to their robustness. To answer to this hypothesis, we designed an empirical study with neural network models obtained through random graphs used as sparse structural priors for the networks. We additionally investigated the evaluation of a randomly pruned fully connected network as a point of reference. We found that robustness measures are independent of initialization methods but show weak correlations with graph properties: higher graph densities correlate with lower robustness, but higher average path lengths and average node eccentricities show negative correlations with robustness measures. We hope to motivate further empirical and analytical research to tightening an answer to our hypothesis.