Clustering with Outlier Removal

TL;DR

This paper introduces the COR algorithm that jointly performs clustering and outlier detection by transforming data into a binary space, optimizing cluster compactness while removing outliers, and demonstrating superior results over existing methods.

Contribution

The paper proposes a novel unified algorithm, COR, that effectively combines clustering and outlier detection using a binary space transformation and K-means optimization.

Findings

COR outperforms state-of-the-art methods in cluster validity.

COR effectively detects outliers across various datasets.

The algorithm demonstrates practical utility in real-world scenarios like flight trajectory analysis.

Abstract

Cluster analysis and outlier detection are strongly coupled tasks in data mining area. Cluster structure can be easily destroyed by few outliers; on the contrary, outliers are defined by the concept of cluster, which are recognized as the points belonging to none of the clusters. Unfortunately, most existing studies do not notice the coupled relationship between these two task and handle them separately. In light of this, we consider the joint cluster analysis and outlier detection problem, and propose the Clustering with Outlier Removal (COR) algorithm. Generally speaking, the original space is transformed into the binary space via generating basic partitions in order to define clusters. Then an objective function based Holoentropy is designed to enhance the compactness of each cluster with a few outliers removed. With further analyses on the objective function, only partial of the problem can be handled by K-means optimization. To provide an integrated solution, an auxiliary binary matrix is nontrivally introduced so that COR completely and efficiently solves the challenging problem via a unified K-means-- with theoretical supports. Extensive experimental results on numerous data sets in various domains demonstrate the effectiveness and efficiency of COR significantly over state-of-the-art methods in terms of cluster validity and outlier detection. Some key factors in COR are further analyzed for practical use. Finally, an application on flight trajectory is provided to demonstrate the effectiveness of COR in the real-world scenario.