S-JEA: Stacked Joint Embedding Architectures for Self-Supervised Visual Representation Learning

TL;DR

This paper introduces S-JEA, a stacked joint embedding architecture for self-supervised learning that captures hierarchical semantic concepts, resulting in more interpretable and separable image representations.

Contribution

The work proposes stacking joint embedding architectures to learn hierarchical semantic representations in self-supervised visual learning, enhancing interpretability and concept separability.

Findings

Stacked JEA captures hierarchical semantic concepts.

Performance comparable to traditional JEA with similar parameters.

Visualizations validate semantic hierarchies in learned representations.

Abstract

The recent emergence of Self-Supervised Learning (SSL) as a fundamental paradigm for learning image representations has, and continues to, demonstrate high empirical success in a variety of tasks. However, most SSL approaches fail to learn embeddings that capture hierarchical semantic concepts that are separable and interpretable. In this work, we aim to learn highly separable semantic hierarchical representations by stacking Joint Embedding Architectures (JEA) where higher-level JEAs are input with representations of lower-level JEA. This results in a representation space that exhibits distinct sub-categories of semantic concepts (e.g., model and colour of vehicles) in higher-level JEAs. We empirically show that representations from stacked JEA perform on a similar level as traditional JEA with comparative parameter counts and visualise the representation spaces to validate the semantic hierarchies.