Exploring the Properties and Evolution of Neural Network Eigenspaces during Training

TL;DR

This paper investigates neural network eigenspaces during training, revealing how problem difficulty and capacity influence information processing and how saturation patterns stabilize early, enabling faster analysis.

Contribution

It introduces a method to analyze neural eigenspaces using probes and saturation metrics, highlighting their relation to network capacity and training dynamics.

Findings

Saturation patterns converge early in training

Problem difficulty and capacity affect predictive performance antagonistically

Saturation analysis can detect over- and under-parameterization

Abstract

In this work we explore the information processing inside neural networks using logistic regression probes \cite{probes} and the saturation metric \cite{featurespace_saturation}. We show that problem difficulty and neural network capacity affect the predictive performance in an antagonistic manner, opening the possibility of detecting over- and under-parameterization of neural networks for a given task. We further show that the observed effects are independent from previously reported pathological patterns like the ``tail pattern'' described in \cite{featurespace_saturation}. Finally we are able to show that saturation patterns converge early during training, allowing for a quicker cycle time during analysis