Equivalence in Deep Neural Networks via Conjugate Matrix Ensembles

TL;DR

This paper introduces a numerical method using Mixed Matrix Ensembles and conjugate circular ensembles to detect equivalence between deep neural network architectures, supported by empirical spectral density analysis.

Contribution

It presents a novel spectral density-based approach for establishing neural network equivalences, applicable to both artificial and biological architectures.

Findings

Spectral density differences vanish with specific decay rates.

Cumulative Circular Spectral Difference (CSD) can identify architecture equivalences.

Method applicable to a wide range of vision architectures and biological networks.

Abstract

A numerical approach is developed for detecting the equivalence of deep learning architectures. The method is based on generating Mixed Matrix Ensembles (MMEs) out of deep neural network weight matrices and {\it conjugate circular ensemble} matching the neural architecture topology. Following this, the empirical evidence supports the {\it phenomenon} that difference between spectral densities of neural architectures and corresponding {\it conjugate circular ensemble} are vanishing with different decay rates at the long positive tail part of the spectrum i.e., cumulative Circular Spectral Difference (CSD). This finding can be used in establishing equivalences among different neural architectures via analysis of fluctuations in CSD. We investigated this phenomenon for a wide range of deep learning vision architectures and with circular ensembles originating from statistical quantum mechanics. Practical implications of the proposed method for artificial and natural neural architectures discussed such as the possibility of using the approach in Neural Architecture Search (NAS) and classification of biological neural networks.