Strong bounds for large-scale Minimum Sum-of-Squares Clustering

TL;DR

This paper introduces a new divide-and-conquer method to validate solutions for large-scale Minimum Sum-of-Squares Clustering, providing strong bounds and efficient assessment of heuristic solutions with minimal optimality gaps.

Contribution

The paper presents a novel approach combining problem decomposition and heuristics to efficiently estimate optimality gaps in large-scale MSSC problems.

Findings

Achieves optimality gaps below 3% in most large-scale instances.

Demonstrates computational efficiency for large datasets.

Provides a practical validation tool for heuristic clustering solutions.

Abstract

Clustering is a fundamental technique in data analysis and machine learning, used to group similar data points together. Among various clustering methods, the Minimum Sum-of-Squares Clustering (MSSC) is one of the most widely used. MSSC aims to minimize the total squared Euclidean distance between data points and their corresponding cluster centroids. Due to the unsupervised nature of clustering, achieving global optimality is crucial, yet computationally challenging. The complexity of finding the global solution increases exponentially with the number of data points, making exact methods impractical for large-scale datasets. Even obtaining strong lower bounds on the optimal MSSC objective value is computationally prohibitive, making it difficult to assess the quality of heuristic solutions. We address this challenge by introducing a novel method to validate heuristic MSSC solutions through optimality gaps. Our approach employs a divide-and-conquer strategy, decomposing the problem into smaller instances that can be handled by an exact solver. The decomposition is guided by an auxiliary optimization problem, the "anticlustering problem", for which we design an efficient heuristic. Computational experiments demonstrate the effectiveness of the method for large-scale instances, achieving optimality gaps below 3% in most cases while maintaining reasonable computational times. These results highlight the practicality of our approach in assessing feasible clustering solutions for large datasets, bridging a critical gap in MSSC evaluation.