Conformal Prediction based Spectral Clustering

TL;DR

This paper introduces a novel affinity measure for spectral clustering based on conformal prediction, which improves the capture of contextual similarities between data points and enhances clustering performance.

Contribution

It proposes a new affinity measure using non-conformity from conformal prediction, advancing spectral clustering by better capturing neighborhood relationships.

Findings

The non-conformity based affinity improves clustering accuracy.

The method compares favorably with state-of-the-art techniques.

It generalizes the notion of contextual similarity in spectral clustering.

Abstract

Spectral Clustering(SC) is a prominent data clustering technique of recent times which has attracted much attention from researchers. It is a highly data-driven method and makes no strict assumptions on the structure of the data to be clustered. One of the central pieces of spectral clustering is the construction of an affinity matrix based on a similarity measure between data points. The way the similarity measure is defined between data points has a direct impact on the performance of the SC technique. Several attempts have been made in the direction of strengthening the pairwise similarity measure to enhance the spectral clustering. In this work, we have defined a novel affinity measure by employing the concept of non-conformity used in Conformal Prediction(CP) framework. The non-conformity based affinity captures the relationship between neighborhoods of data points and has the power to generalize the notion of contextual similarity. We have shown that this formulation of affinity measure gives good results and compares well with the state of the art methods.