Ensemble Projection Pursuit for General Nonparametric Regression

TL;DR

This paper introduces an ensemble projection pursuit regression (ePPR) method that improves accuracy and efficiency over traditional PPR, RF, and SVM, demonstrating superior performance in regression and classification tasks.

Contribution

The paper proposes an optimal greedy algorithm and ensemble approach for PPR, establishing theoretical consistency and showing ePPR's superior empirical performance.

Findings

ePPR achieves higher accuracy than RF and SVM.

ePPR maintains efficiency by using the whole data for each term.

ePPR's performance rivals or exceeds that of neural networks on small to medium datasets.

Abstract

The projection pursuit regression (PPR) has played an important role in the development of statistics and machine learning. However, when compared to other established methods like random forests (RF) and support vector machines (SVM), PPR has yet to showcase a similar level of accuracy as a statistical learning technique. In this paper, we revisit the estimation of PPR and propose an \textit{optimal} greedy algorithm and an ensemble approach via "feature bagging", hereafter referred to as ePPR, aiming to improve the efficacy. Compared to RF, ePPR has two main advantages. Firstly, its theoretical consistency can be proved for more general regression functions as long as they are $L^2$ integrable, and higher consistency rates can be achieved. Secondly, ePPR does not split the samples, and thus each term of PPR is estimated using the whole data, making the minimization more efficient and guaranteeing the smoothness of the estimator. Extensive comparisons based on real data sets show that ePPR is more efficient in regression and classification than RF and other competitors. The efficacy of ePPR, a variant of Artificial Neural Networks (ANN), demonstrates that with suitable statistical tuning, ANN can equal or even exceed RF in dealing with small to medium-sized datasets. This revelation challenges the widespread belief that ANN's superiority over RF is limited to processing extensive sample sizes.