Near-optimal Linear Predictive Clustering in Non-separable Spaces via MIP and QPBO Reductions

TL;DR

This paper introduces scalable, near-optimal methods for linear predictive clustering in non-separable spaces by reducing complex optimization problems to more efficient formulations, improving accuracy and scalability.

Contribution

It develops two novel approaches that approximate LPC efficiently using MIP and QPBO reductions, with theoretical error bounds and improved scalability.

Findings

Achieves near-optimal solutions with lower regression errors.

Demonstrates superior scalability over existing MIP methods.

Provides effective approximations in synthetic and real-world datasets.

Abstract

Linear Predictive Clustering (LPC) partitions samples based on shared linear relationships between feature and target variables, with numerous applications including marketing, medicine, and education. Greedy optimization methods, commonly used for LPC, alternate between clustering and linear regression but lack global optimality. While effective for separable clusters, they struggle in non-separable settings where clusters overlap in feature space. In an alternative constrained optimization paradigm, Bertsimas and Shioda (2007) formulated LPC as a Mixed-Integer Program (MIP), ensuring global optimality regardless of separability but suffering from poor scalability. This work builds on the constrained optimization paradigm to introduce two novel approaches that improve the efficiency of global optimization for LPC. By leveraging key theoretical properties of separability, we derive near-optimal approximations with provable error bounds, significantly reducing the MIP formulation's complexity and improving scalability. Additionally, we can further approximate LPC as a Quadratic Pseudo-Boolean Optimization (QPBO) problem, achieving substantial computational improvements in some settings. Comparative analyses on synthetic and real-world datasets demonstrate that our methods consistently achieve near-optimal solutions with substantially lower regression errors than greedy optimization while exhibiting superior scalability over existing MIP formulations.