Self-Labeling Refinement for Robust Representation Learning with Bootstrap Your Own Latent

TL;DR

This paper investigates the role of Batch Normalisation in BYOL, finds it is unnecessary, and introduces new loss functions to leverage semantically similar pairs, improving representation learning performance.

Contribution

It demonstrates that Batch Normalisation is not essential for BYOL and proposes two novel loss functions to enhance the learning process by utilizing semantically similar pairs.

Findings

BYOL does not require Batch Normalisation.

CCSL loss improves BYOL performance to 76.87%.

CSSL loss performs comparably to vanilla BYOL.

Abstract

In this work, we have worked towards two major goals. Firstly, we have investigated the importance of Batch Normalisation (BN) layers in a non-contrastive representation learning framework called Bootstrap Your Own Latent (BYOL). We conducted several experiments to conclude that BN layers are not necessary for representation learning in BYOL. Moreover, BYOL only learns from the positive pairs of images but ignores other semantically similar images in the same input batch. For the second goal, we have introduced two new loss functions to determine the semantically similar pairs in the same input batch of images and reduce the distance between their representations. These loss functions are Cross-Cosine Similarity Loss (CCSL) and Cross-Sigmoid Similarity Loss (CSSL). Using the proposed loss functions, we are able to surpass the performance of Vanilla BYOL (71.04%) by training the BYOL framework using CCSL loss (76.87%) on the STL10 dataset. BYOL trained using CSSL loss performs comparably with Vanilla BYOL.