Information Theoretical Importance Sampling Clustering

TL;DR

This paper introduces an information theoretical importance sampling clustering method (ITISC) that addresses distribution deviations in clustering, validated on synthetic and real-world datasets, and relates to fuzzy c-means.

Contribution

It proposes a novel importance sampling based clustering approach that minimizes worst-case expected distortions under distribution deviation constraints.

Findings

Effective on synthetic datasets and load forecasting tasks.

Reveals fuzzy c-means as a special case of ITISC.

Provides a physical interpretation for fuzzy exponent m.

Abstract

A current assumption of most clustering methods is that the training data and future data are taken from the same distribution. However, this assumption may not hold in most real-world scenarios. In this paper, we propose an information theoretical importance sampling based approach for clustering problems (ITISC) which minimizes the worst case of expected distortions under the constraint of distribution deviation. The distribution deviation constraint can be converted to the constraint over a set of weight distributions centered on the uniform distribution derived from importance sampling. The objective of the proposed approach is to minimize the loss under maximum degradation hence the resulting problem is a constrained minimax optimization problem which can be reformulated to an unconstrained problem using the Lagrange method. The optimization problem can be solved by both an alternative optimization algorithm or a general optimization routine by commercially available software. Experiment results on synthetic datasets and a real-world load forecasting problem validate the effectiveness of the proposed model. Furthermore, we show that fuzzy c-means is a special case of ITISC with the logarithmic distortion, and this observation provides an interesting physical interpretation for fuzzy exponent $m$.