Classifying the classifier: dissecting the weight space of neural networks

TL;DR

This study explores the high-dimensional weight space of neural networks by training meta-classifiers to identify training setup properties from weight patterns, offering new insights into model interpretability and a large dataset for future research.

Contribution

It introduces a novel approach using deep meta-classifiers to analyze neural weight space and provides a large dataset for further exploration.

Findings

Meta-classifiers can identify training hyper-parameters from weight patterns.

Weight space contains discernible signatures of training variations.

The approach enhances explainability of neural network training processes.

Abstract

This paper presents an empirical study on the weights of neural networks, where we interpret each model as a point in a high-dimensional space -- the neural weight space. To explore the complex structure of this space, we sample from a diverse selection of training variations (dataset, optimization procedure, architecture, etc.) of neural network classifiers, and train a large number of models to represent the weight space. Then, we use a machine learning approach for analyzing and extracting information from this space. Most centrally, we train a number of novel deep meta-classifiers with the objective of classifying different properties of the training setup by identifying their footprints in the weight space. Thus, the meta-classifiers probe for patterns induced by hyper-parameters, so that we can quantify how much, where, and when these are encoded through the optimization process. This provides a novel and complementary view for explainable AI, and we show how meta-classifiers can reveal a great deal of information about the training setup and optimization, by only considering a small subset of randomly selected consecutive weights. To promote further research on the weight space, we release the neural weight space (NWS) dataset -- a collection of 320K weight snapshots from 16K individually trained deep neural networks.