CEREAL: Few-Sample Clustering Evaluation

TL;DR

CEREAL is a comprehensive framework that improves the estimation of clustering quality with limited labels by combining active sampling, semi-supervised learning, and pseudo-labeling, reducing bias and annotation effort.

Contribution

It introduces novel NMI-based acquisition functions, integrates semi-supervised training of surrogate models, and extends evaluation to pairwise annotations, advancing few-sample clustering evaluation.

Findings

Reduces estimation error by up to 57% compared to baselines.

Effective across vision and language datasets.

Agnostic to clustering algorithms and metrics.

Abstract

Evaluating clustering quality with reliable evaluation metrics like normalized mutual information (NMI) requires labeled data that can be expensive to annotate. We focus on the underexplored problem of estimating clustering quality with limited labels. We adapt existing approaches from the few-sample model evaluation literature to actively sub-sample, with a learned surrogate model, the most informative data points for annotation to estimate the evaluation metric. However, we find that their estimation can be biased and only relies on the labeled data. To that end, we introduce CEREAL, a comprehensive framework for few-sample clustering evaluation that extends active sampling approaches in three key ways. First, we propose novel NMI-based acquisition functions that account for the distinctive properties of clustering and uncertainties from a learned surrogate model. Next, we use ideas from semi-supervised learning and train the surrogate model with both the labeled and unlabeled data. Finally, we pseudo-label the unlabeled data with the surrogate model. We run experiments to estimate NMI in an active sampling pipeline on three datasets across vision and language. Our results show that CEREAL reduces the area under the absolute error curve by up to 57% compared to the best sampling baseline. We perform an extensive ablation study to show that our framework is agnostic to the choice of clustering algorithm and evaluation metric. We also extend CEREAL from clusterwise annotations to pairwise annotations. Overall, CEREAL can efficiently evaluate clustering with limited human annotations.