Domain-Agnostic Clustering with Self-Distillation

TL;DR

This paper introduces a domain-agnostic clustering method based on self-distillation that does not rely on data augmentation, outperforming existing algorithms and enhancing unsupervised learning efficiency.

Contribution

It presents a novel self-distillation algorithm for domain-agnostic clustering that eliminates the need for data augmentation and improves performance over prior methods.

Findings

Outperforms existing domain-agnostic algorithms on CIFAR-10.

Knowledge distillation enhances unsupervised representation learning.

Self-distillation improves convergence of DeepCluster-v2.

Abstract

Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmentation, rendering them ineffective for many learning tasks where insufficient domain knowledge exists for performing augmentation. We propose a new self-distillation based algorithm for domain-agnostic clustering. Our method builds upon the existing deep clustering frameworks and requires no separate student model. The proposed method outperforms existing domain agnostic (augmentation-free) algorithms on CIFAR-10. We empirically demonstrate that knowledge distillation can improve unsupervised representation learning by extracting richer `dark knowledge' from the model than using predicted labels alone. Preliminary experiments also suggest that self-distillation improves the convergence of DeepCluster-v2.