A Fast and Effective Method for Euclidean Anticlustering: The Assignment-Based-Anticlustering Algorithm

TL;DR

This paper introduces the Assignment-Based Anticlustering (ABA) algorithm, a scalable and efficient method for Euclidean anticlustering that outperforms existing heuristics in large-scale datasets and applications.

Contribution

The paper presents a novel ABA algorithm that significantly improves scalability and solution quality for Euclidean anticlustering tasks, especially on large datasets.

Findings

ABA outperforms fast_anticlustering in solution quality and speed.

ABA scales to datasets with millions of objects and hundreds of thousands of anticlusters.

ABA surpasses METIS in both solution quality and runtime for K-cut partitioning.

Abstract

The anticlustering problem is to partition a set of objects into K equal-sized anticlusters such that the sum of distances within anticlusters is maximized. The anticlustering problem is NP-hard. We focus on anticlustering in Euclidean spaces, where the input data is tabular and each object is represented as a D-dimensional feature vector. Distances are measured as squared Euclidean distances between the respective vectors. Applications of Euclidean anticlustering include social studies, particularly in psychology, K-fold cross-validation in which each fold should be a good representative of the entire dataset, the creation of mini-batches for gradient descent in neural network training, and balanced K-cut partitioning. In particular, machine-learning applications involve million-scale datasets and very large values of K, making scalable anticlustering algorithms essential. Existing algorithms are either exact methods that can solve only small instances or heuristic methods, among which the most scalable is the exchange-based heuristic fast_anticlustering. We propose a new algorithm, the Assignment-Based Anticlustering algorithm (ABA), which scales to very large instances. A computational study shows that ABA outperforms fast_anticlustering in both solution quality and running time. Moreover, ABA scales to instances with millions of objects and hundreds of thousands of anticlusters within short running times, beyond what fast_anticlustering can handle. As a balanced K-cut partitioning method for tabular data, ABA is superior to the well-known METIS method in both solution quality and running time. The code of the ABA algorithm is available on GitHub.