Large Deviation Analysis of Function Sensitivity in Random Deep Neural Networks

TL;DR

This paper applies large deviation theory and path integral analysis to study how finite-size deep neural networks deviate from mean field solutions, focusing on robustness to parameter perturbations like sparsification and binarization.

Contribution

It introduces a novel application of large deviation theory to analyze finite-size effects and function sensitivity in deep neural networks with different activation functions.

Findings

ReLU networks are more robust to parameter perturbations than sign activation networks.

Finite-size deviations from mean field solutions are quantitatively characterized.

Robustness differences reflect the simplicity of functions generated by ReLU versus sign activations.

Abstract

Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The parameter perturbations investigated include weight sparsification (dilution) and binarization, which are commonly used in model simplification, for both ReLU and sign activation functions. We find that random networks with ReLU activation are more robust to parameter perturbations with respect to their counterparts with sign activation, which arguably is reflected in the simplicity of the functions they generate.