Learning with Subset Stacking

TL;DR

The paper introduces LESS, a novel regression algorithm that models heterogeneous relationships by creating local predictors on data subsets and combining them, demonstrating competitive performance against existing methods.

Contribution

It presents a new local regression method called LESS that uses subset stacking, with bagging and boosting variants, for improved modeling of heterogeneous data.

Findings

LESS is highly competitive with state-of-the-art methods.

The algorithm effectively captures heterogeneity in data.

Bagging and boosting variants enhance performance.

Abstract

We propose a new regression algorithm that learns from a set of input-output pairs. Our algorithm is designed for populations where the relation between the input variables and the output variable exhibits a heterogeneous behavior across the predictor space. The algorithm starts with generating subsets that are concentrated around random points in the input space. This is followed by training a local predictor for each subset. Those predictors are then combined in a novel way to yield an overall predictor. We call this algorithm "LEarning with Subset Stacking" or LESS, due to its resemblance to the method of stacking regressors. We offer bagging and boosting variants of LESS and test against the state-of-the-art methods on several datasets. Our comparison shows that LESS is highly competitive.