Exploring Learned Representations of Neural Networks with Principal Component Analysis

TL;DR

This paper uses PCA to analyze layer-wise representations in ResNet-18, revealing that a small subset of principal components can explain classifier performance and providing insights into neural collapse phenomena.

Contribution

It introduces a PCA-based analysis revealing that few principal components suffice for high classification accuracy and offers surrogate models to interpret neural collapse in DNNs.

Findings

20% of variance suffices for high accuracy in some layers

First ~100 PCs determine classifier performance

Surrogate models help locate neural collapse onset

Abstract

Understanding feature representation for deep neural networks (DNNs) remains an open question within the general field of explainable AI. We use principal component analysis (PCA) to study the performance of a k-nearest neighbors classifier (k-NN), nearest class-centers classifier (NCC), and support vector machines on the learned layer-wise representations of a ResNet-18 trained on CIFAR-10. We show that in certain layers, as little as 20% of the intermediate feature-space variance is necessary for high-accuracy classification and that across all layers, the first ~100 PCs completely determine the performance of the k-NN and NCC classifiers. We relate our findings to neural collapse and provide partial evidence for the related phenomenon of intermediate neural collapse. Our preliminary work provides three distinct yet interpretable surrogate models for feature representation with an affine linear model the best performing. We also show that leveraging several surrogate models affords us a clever method to estimate where neural collapse may initially occur within the DNN.