Learning Feature Nonlinearities with Non-Convex Regularized Binned Regression

TL;DR

This paper introduces a novel non-convex regularized binned regression method that efficiently learns feature nonlinearities, outperforming traditional models in large-scale, complex problems with high nonlinearity.

Contribution

It proposes a new algorithm combining feature binning, non-convex regularization, and smoothness constraints, with proven statistical and computational efficiency.

Findings

Achieves linear convergence rate.

Performs competitively with state-of-the-art methods.

Accurately models feature nonlinearities in diverse datasets.

Abstract

For various applications, the relations between the dependent and independent variables are highly nonlinear. Consequently, for large scale complex problems, neural networks and regression trees are commonly preferred over linear models such as Lasso. This work proposes learning the feature nonlinearities by binning feature values and finding the best fit in each quantile using non-convex regularized linear regression. The algorithm first captures the dependence between neighboring quantiles by enforcing smoothness via piecewise-constant/linear approximation and then selects a sparse subset of good features. We prove that the proposed algorithm is statistically and computationally efficient. In particular, it achieves linear rate of convergence while requiring near-minimal number of samples. Evaluations on synthetic and real datasets demonstrate that algorithm is competitive with current state-of-the-art and accurately learns feature nonlinearities. Finally, we explore an interesting connection between the binning stage of our algorithm and sparse Johnson-Lindenstrauss matrices.