Normalized Space Alignment: A Versatile Metric for Representation Analysis

TL;DR

Normalized Space Alignment (NSA) is a versatile, efficient metric for analyzing, comparing, and aligning neural network representations across layers and models, applicable as an analysis tool and a loss function.

Contribution

NSA introduces a novel manifold analysis technique that compares pairwise distances to evaluate and align neural representations, functioning as both a similarity metric and a differentiable loss.

Findings

NSA effectively compares neural representations across layers and models.

NSA can serve as a loss function to improve model alignment.

NSA is computationally efficient and suitable for mini-batch training.

Abstract

We introduce a manifold analysis technique for neural network representations. Normalized Space Alignment (NSA) compares pairwise distances between two point clouds derived from the same source and having the same size, while potentially possessing differing dimensionalities. NSA can act as both an analytical tool and a differentiable loss function, providing a robust means of comparing and aligning representations across different layers and models. It satisfies the criteria necessary for both a similarity metric and a neural network loss function. We showcase NSA's versatility by illustrating its utility as a representation space analysis metric, a structure-preserving loss function, and a robustness analysis tool. NSA is not only computationally efficient but it can also approximate the global structural discrepancy during mini-batching, facilitating its use in a wide variety of neural network training paradigms.