A New Framework to Analyse the Distributional Robustness of Deep Neural Networks

TL;DR

This paper introduces a novel framework for analyzing the distributional robustness of deep neural networks by examining layer interactions and class separation metrics, validated on CIFAR-10 and ImageNet.

Contribution

The authors propose a new diagnostic framework using Bernoulli models to assess neural network robustness and distinguish memorized from generalizable models.

Findings

Metrics can differentiate memorized and non-memorized networks.

Distribution shifts decrease class separation in the proposed diagnostics.

Activation space properties do not replicate the same robustness indicators.

Abstract

Deep neural networks have achieved impressive performance on a variety of tasks, but their brittleness to distributional shifts remains a significant barrier to real-world deployment. In this paper, we propose a framework to analyse and quantify the distributional robustness of neural networks by studying the interactions between layer weights and activations. We model these interactions using Bernoulli distributions, using the separation between classes as a diagnostic proxy for robustness. We demonstrate the usefulness of this framework through models trained on CIFAR-10 and ImageNet. We show that our proposed metrics can distinguish between networks that have memorised their training data and those that have not. We also perform analogous experiments in the activation space and find that the same properties do not hold up. Additionally, we investigate the behaviour of our metrics under various distribution shifts and show that these shifts reduce separation under our path-based diagnostics. Our results suggest that this framework provides useful model-level diagnostics of representation structure and robustness.