Over-parameterization and Adversarial Robustness in Neural Networks: An Overview and Empirical Analysis

TL;DR

This paper provides an empirical analysis of over-parameterized neural networks, demonstrating their robustness against adversarial attacks and emphasizing the importance of attack reliability in evaluating model robustness.

Contribution

It offers a comprehensive empirical study on over-parameterized networks' robustness and assesses the reliability of adversarial attack methods used in evaluations.

Findings

Over-parameterized networks are more robust than under-parameterized ones.

The reliability of adversarial attacks significantly impacts robustness evaluation.

Previous contradictory results may stem from unreliable attack methods.

Abstract

Thanks to their extensive capacity, over-parameterized neural networks exhibit superior predictive capabilities and generalization. However, having a large parameter space is considered one of the main suspects of the neural networks' vulnerability to adversarial example -- input samples crafted ad-hoc to induce a desired misclassification. Relevant literature has claimed contradictory remarks in support of and against the robustness of over-parameterized networks. These contradictory findings might be due to the failure of the attack employed to evaluate the networks' robustness. Previous research has demonstrated that depending on the considered model, the algorithm employed to generate adversarial examples may not function properly, leading to overestimating the model's robustness. In this work, we empirically study the robustness of over-parameterized networks against adversarial examples. However, unlike the previous works, we also evaluate the considered attack's reliability to support the results' veracity. Our results show that over-parameterized networks are robust against adversarial attacks as opposed to their under-parameterized counterparts.