On Interpretable Approaches to Cluster, Classify and Represent Multi-Subspace Data via Minimum Lossy Coding Length based on Rate-Distortion Theory

TL;DR

This paper presents three interpretable, information-theoretic methods for clustering, classifying, and representing high-dimensional data based on rate-distortion theory, suitable for finite-sample, mixed Gaussian data.

Contribution

It introduces novel, theoretically grounded algorithms for clustering, classification, and representation using lossy coding length criteria derived from rate-distortion theory.

Findings

Effective for finite-sample, sparse, or degenerate data

Suitable for mixed Gaussian distributions or subspaces

Provides a theoretical guide for white-box machine learning

Abstract

To cluster, classify and represent are three fundamental objectives of learning from high-dimensional data with intrinsic structure. To this end, this paper introduces three interpretable approaches, i.e., segmentation (clustering) via the Minimum Lossy Coding Length criterion, classification via the Minimum Incremental Coding Length criterion and representation via the Maximal Coding Rate Reduction criterion. These are derived based on the lossy data coding and compression framework from the principle of rate distortion in information theory. These algorithms are particularly suitable for dealing with finite-sample data (allowed to be sparse or almost degenerate) of mixed Gaussian distributions or subspaces. The theoretical value and attractive features of these methods are summarized by comparison with other learning methods or evaluation criteria. This summary note aims to provide a theoretical guide to researchers (also engineers) interested in understanding 'white-box' machine (deep) learning methods.