Curvature-based Comparison of Two Neural Networks

TL;DR

This paper introduces a geometric approach to compare two neural networks by analyzing the curvature of manifolds formed by their activation vectors, revealing intrinsic similarities and differences.

Contribution

It proposes a new data generation method and a systematic strategy using Riemann and sectional curvature to compare neural network feature manifolds.

Findings

Identifies geometric properties reflecting network similarities

Demonstrates differences in feature extraction mechanisms

Provides insights into the intrinsic structure of neural networks

Abstract

In this paper we show the similarities and differences of two deep neural networks by comparing the manifolds composed of activation vectors in each fully connected layer of them. The main contribution of this paper includes 1) a new data generating algorithm which is crucial for determining the dimension of manifolds; 2) a systematic strategy to compare manifolds. Especially, we take Riemann curvature and sectional curvature as part of criterion, which can reflect the intrinsic geometric properties of manifolds. Some interesting results and phenomenon are given, which help in specifying the similarities and differences between the features extracted by two networks and demystifying the intrinsic mechanism of deep neural networks.