Complexity of Linear Regions in Self-supervised Deep ReLU Networks

TL;DR

This paper analyzes the complexity of linear regions in self-supervised ReLU networks, revealing how different SSL methods influence geometric properties and correlating these with representation quality.

Contribution

It introduces a geometric analysis of linear regions in SSL models, comparing various training methods and linking polytopal metrics to model performance.

Findings

Self-supervised methods create fewer regions for similar accuracy.

Contrastive methods rapidly expand regions during training.

Polytopal metrics can indicate representation quality and model performance.

Abstract

There has been growing interest in studying the complexity of Rectified Linear Unit (ReLU) based activation networks. Recent work investigates the evolution of the number of piecewise-linear partitions (linear regions) that are formed during training. However, current research is limited to examining the complexity of models trained in a supervised way. Self-Supervised Learning (SSL) differs in that it directly optimises the representation space using a loss function to enhance the model's performance across multiple downstream tasks. This study investigates the local distribution of linear regions produced by SSL models. We demonstrate that the evolution of linear regions correlates with the representation quality by utilising SplineCam to extract two-dimensional polytopes near the data distribution. We track the number, area, eccentricity, and boundaries of regions throughout training. The study compares supervised, contrastive, and self-distillation methods over two standard benchmark datasets, MNIST and FashionMNIST. The analysis of the experimental results shows that self-supervised methods create substantially fewer regions to achieve comparable accuracy to supervised models. Contrastive methods rapidly expand regions over time, whereas self-distillation methods tend to consolidate by merging neighbouring regions. Lastly, we can detect representation collapse early within the geometric space of linear regions. Our analysis suggests that polytopal metrics can serve as reliable indicators of representation quality and model performance.