Learning with Clustering Structure

TL;DR

This paper introduces a unified approach to supervised learning with clustering constraints on features or samples, improving interpretability and prediction, with algorithms based on k-means clustering and extensions to sparse clustering.

Contribution

It develops a unified optimization framework for clustering-based constraints in supervised learning, with efficient algorithms and convergence proofs, applicable to feature and sample clustering.

Findings

Algorithms perform efficiently with k-means approximation.

Methods are robust on artificial and real datasets.

Extensions incorporate sparsity into clustering constraints.

Abstract

We study supervised learning problems using clustering constraints to impose structure on either features or samples, seeking to help both prediction and interpretation. The problem of clustering features arises naturally in text classification for instance, to reduce dimensionality by grouping words together and identify synonyms. The sample clustering problem on the other hand, applies to multiclass problems where we are allowed to make multiple predictions and the performance of the best answer is recorded. We derive a unified optimization formulation highlighting the common structure of these problems and produce algorithms whose core iteration complexity amounts to a k-means clustering step, which can be approximated efficiently. We extend these results to combine sparsity and clustering constraints, and develop a new projection algorithm on the set of clustered sparse vectors. We prove convergence of our algorithms on random instances, based on a union of subspaces interpretation of the clustering structure. Finally, we test the robustness of our methods on artificial data sets as well as real data extracted from movie reviews.