PET-TURTLE: Deep Unsupervised Support Vector Machines for Imbalanced Data Clusters

TL;DR

PET-TURTLE is a novel deep clustering algorithm that effectively handles imbalanced data by incorporating a power law prior and sparse logits, improving accuracy and cluster balance in unsupervised learning.

Contribution

It extends TURTLE by introducing a new cost function for imbalanced data and sparse logits, enhancing deep clustering performance on imbalanced datasets.

Findings

Improves clustering accuracy on imbalanced datasets.

Reduces over-prediction of minority clusters.

Enhances overall clustering quality.

Abstract

Foundation vision, audio, and language models enable zero-shot performance on downstream tasks via their latent representations. Recently, unsupervised learning of data group structure with deep learning methods has gained popularity. TURTLE, a state of the art deep clustering algorithm, uncovers data labeling without supervision by alternating label and hyperplane updates, maximizing the hyperplane margin, in a similar fashion to support vector machines (SVMs). However, TURTLE assumes clusters are balanced; when data is imbalanced, it yields non-ideal hyperplanes that cause higher clustering error. We propose PET-TURTLE, which generalizes the cost function to handle imbalanced data distributions by a power law prior. Additionally, by introducing sparse logits in the labeling process, PET-TURTLE optimizes a simpler search space that in turn improves accuracy for balanced datasets. Experiments on synthetic and real data show that PET-TURTLE improves accuracy for imbalanced sources, prevents over-prediction of minority clusters, and enhances overall clustering.