Locally Near Optimal Piecewise Linear Regression in High Dimensions via Difference of Max-Affine Functions

TL;DR

This paper introduces a new algorithm, ABGD, for piecewise linear regression in high dimensions, with theoretical guarantees and competitive empirical performance, by parametrizing functions as differences of max-affine functions.

Contribution

The paper develops ABGD, a parametric method with local convergence analysis for piecewise linear regression using difference of max-affine functions, achieving near-optimal sample complexity.

Findings

ABGD converges linearly to an $ ext{epsilon}$-accurate estimate with $ ilde{O}(d imes ext{max}(rac{\sigma_z}{ ext{epsilon}},1)^2)$ samples.

Exact recovery is possible with $ ilde{O}(d)$ samples in noiseless settings.

Synthetic and real-world experiments validate the theoretical guarantees and competitive performance.

Abstract

This paper presents a parametric solution to piecewise linear regression through the Adaptive Block Gradient Descent (ABGD) algorithm. The heart of the method is the parametrization of piecewise linear functions as the difference of max-affine (DoMA) functions. A non-asymptotic local convergence analysis for ABGD is provided under sub-Gaussian covariate and noise distributions. To initialize ABGD, we adapt a prior algorithm originally developed for the simpler setting of max-affine functions. When suitably initialized, ABGD converges linearly to an $\epsilon$-accurate estimate given $\tilde{\mathcal{O}}(d\max(\sigma_z/\epsilon,1)^2)$ observations where $\sigma_z^2$ denotes the noise variance. This implies exact recovery given $\tilde{\mathcal{O}}(d)$ samples in the noiseless case. Also, such a rate is shown to be minimax optimal up to logarithmic factors. Synthetic numerical results corroborate the theoretical guarantees for ABGD. We also observe competitive performance compared to the state-of-the-art methods on real-world datasets.