Ladder Siamese Network: a Method and Insights for Multi-level Self-Supervised Learning

TL;DR

The paper introduces the Ladder Siamese Network, a self-supervised learning framework that leverages multi-level intermediate self-supervisions to improve training stability and performance across image classification, detection, and segmentation tasks.

Contribution

It reveals the effectiveness of multi-level self-supervisions in non-contrastive Siamese frameworks through theoretical and empirical analysis, enhancing various vision tasks.

Findings

Improves ImageNet linear classification by 1.0% points

Enhances COCO detection by 1.2% points

Boosts PASCAL VOC segmentation by 3.1% points

Abstract

Siamese-network-based self-supervised learning (SSL) suffers from slow convergence and instability in training. To alleviate this, we propose a framework to exploit intermediate self-supervisions in each stage of deep nets, called the Ladder Siamese Network. Our self-supervised losses encourage the intermediate layers to be consistent with different data augmentations to single samples, which facilitates training progress and enhances the discriminative ability of the intermediate layers themselves. While some existing work has already utilized multi-level self supervisions in SSL, ours is different in that 1) we reveal its usefulness with non-contrastive Siamese frameworks in both theoretical and empirical viewpoints, and 2) ours improves image-level classification, instance-level detection, and pixel-level segmentation simultaneously. Experiments show that the proposed framework can improve BYOL baselines by 1.0% points in ImageNet linear classification, 1.2% points in COCO detection, and 3.1% points in PASCAL VOC segmentation. In comparison with the state-of-the-art methods, our Ladder-based model achieves competitive and balanced performances in all tested benchmarks without causing large degradation in one.