Self-supervised Label Augmentation via Input Transformations

TL;DR

This paper introduces a self-supervised label augmentation method that improves model accuracy by jointly learning original and augmented tasks through input transformations, applicable even in fully-labeled datasets.

Contribution

It proposes a unified training scheme that combines original and self-supervised tasks via label augmentation, enhancing accuracy and enabling efficient inference through self-distillation.

Findings

Significant accuracy improvements across various datasets.

Effective in few-shot and imbalanced classification scenarios.

Enables aggregated inference and faster self-distillation.

Abstract

Self-supervised learning, which learns by constructing artificial labels given only the input signals, has recently gained considerable attention for learning representations with unlabeled datasets, i.e., learning without any human-annotated supervision. In this paper, we show that such a technique can be used to significantly improve the model accuracy even under fully-labeled datasets. Our scheme trains the model to learn both original and self-supervised tasks, but is different from conventional multi-task learning frameworks that optimize the summation of their corresponding losses. Our main idea is to learn a single unified task with respect to the joint distribution of the original and self-supervised labels, i.e., we augment original labels via self-supervision of input transformation. This simple, yet effective approach allows to train models easier by relaxing a certain invariant constraint during learning the original and self-supervised tasks simultaneously. It also enables an aggregated inference which combines the predictions from different augmentations to improve the prediction accuracy. Furthermore, we propose a novel knowledge transfer technique, which we refer to as self-distillation, that has the effect of the aggregated inference in a single (faster) inference. We demonstrate the large accuracy improvement and wide applicability of our framework on various fully-supervised settings, e.g., the few-shot and imbalanced classification scenarios.