SigCLR: Sigmoid Contrastive Learning of Visual Representations

TL;DR

SigCLR introduces a sigmoid contrastive learning method using logistic loss for visual representations, achieving competitive results without requiring a global view, and emphasizing the importance of learnable bias.

Contribution

The paper presents SigCLR, a novel contrastive learning approach that replaces cross-entropy with logistic loss, simplifying the training process and maintaining competitive performance.

Findings

SigCLR performs well on CIFAR-10, CIFAR-100, and Tiny-IN datasets.

Learnable bias is crucial for the effectiveness of SigCLR.

Fixed temperature is necessary for optimal performance.

Abstract

We propose SigCLR: Sigmoid Contrastive Learning of Visual Representations. SigCLR utilizes the logistic loss that only operates on pairs and does not require a global view as in the cross-entropy loss used in SimCLR. We show that logistic loss shows competitive performance on CIFAR-10, CIFAR-100, and Tiny-IN compared to other established SSL objectives. Our findings verify the importance of learnable bias as in the case of SigLUP, however, it requires a fixed temperature as in the SimCLR to excel. Overall, SigCLR is a promising replacement for the SimCLR which is ubiquitous and has shown tremendous success in various domains.