Understanding Weight Similarity of Neural Networks via Chain Normalization Rule and Hypothesis-Training-Testing

TL;DR

This paper introduces a novel weight similarity measure for neural networks using chain normalization and an extended hypothesis-testing approach, revealing that models trained with SGD tend to converge to similar local solutions across different architectures.

Contribution

It proposes a new weight similarity measure based on chain normalization and hypothesis-training-testing, providing insights into neural network local solutions.

Findings

Weights of identical networks trained with SGD converge to similar solutions

The proposed measure effectively quantifies weight similarity across architectures

Experiments validate the hypothesis of weight similarity in neural networks

Abstract

We present a weight similarity measure method that can quantify the weight similarity of non-convex neural networks. To understand the weight similarity of different trained models, we propose to extract the feature representation from the weights of neural networks. We first normalize the weights of neural networks by introducing a chain normalization rule, which is used for weight representation learning and weight similarity measure. We extend the traditional hypothesis-testing method to a hypothesis-training-testing statistical inference method to validate the hypothesis on the weight similarity of neural networks. With the chain normalization rule and the new statistical inference, we study the weight similarity measure on Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN), and find that the weights of an identical neural network optimized with the Stochastic Gradient Descent (SGD) algorithm converge to a similar local solution in a metric space. The weight similarity measure provides more insight into the local solutions of neural networks. Experiments on several datasets consistently validate the hypothesis of weight similarity measure.