DUEL: Adaptive Duplicate Elimination on Working Memory for Self-Supervised Learning

TL;DR

This paper introduces DUEL, a novel self-supervised learning framework that adaptively eliminates duplicates in working memory to improve stability and performance in the presence of data collisions and class imbalance.

Contribution

The paper proposes a new SSL method with adaptive duplicate elimination inspired by human working memory, addressing data collisions and class imbalance issues.

Findings

Improved stability in SSL with duplicate elimination.

Enhanced performance on downstream tasks.

Effective handling of class imbalance in real-world data.

Abstract

In Self-Supervised Learning (SSL), it is known that frequent occurrences of the collision in which target data and its negative samples share the same class can decrease performance. Especially in real-world data such as crawled data or robot-gathered observations, collisions may occur more often due to the duplicates in the data. To deal with this problem, we claim that sampling negative samples from the adaptively debiased distribution in the memory makes the model more stable than sampling from a biased dataset directly. In this paper, we introduce a novel SSL framework with adaptive Duplicate Elimination (DUEL) inspired by the human working memory. The proposed framework successfully prevents the downstream task performance from degradation due to a dramatic inter-class imbalance.