Simplicial Embeddings in Self-Supervised Learning and Downstream Classification

TL;DR

This paper introduces Simplicial Embeddings (SEM) in self-supervised learning, which improve generalization and semantic coherence in downstream classification tasks on natural image datasets.

Contribution

It formally proves SEM's theoretical advantages and empirically demonstrates improved generalization and semantic feature grouping in image classification.

Findings

SEM leads to better generalization than unnormalized representations.

SSL methods with SEM outperform baselines on CIFAR-100 and ImageNet.

SEM features show emergent semantic coherence in classification.

Abstract

Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into $L$ simplices of $V$ dimensions each using a softmax operation. This procedure conditions the representation onto a constrained space during pretraining and imparts an inductive bias for group sparsity. For downstream classification, we formally prove that the SEM representation leads to better generalization than an unnormalized representation. Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR-100 and ImageNet. Finally, when used in a downstream classification task, we show that SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes.