Bagged Polynomial Regression and Neural Networks

TL;DR

This paper introduces bagged polynomial regression (BPR), a more interpretable alternative to neural networks that achieves comparable accuracy through improved convergence rates and partitioning strategies, demonstrated on benchmark and satellite data.

Contribution

The paper develops new convergence rate results for polynomial estimators and proposes BPR, combining bagging and partitioning to enhance performance and interpretability.

Findings

BPR matches neural network accuracy on MNIST.

Partitioning improves convergence rates in smooth settings.

BPR performs well in satellite crop classification.

Abstract

Series and polynomial regression are able to approximate the same function classes as neural networks. However, these methods are rarely used in practice, although they offer more interpretability than neural networks. In this paper, we show that a potential reason for this is the slow convergence rate of polynomial regression estimators and propose the use of \textit{bagged} polynomial regression (BPR) as an attractive alternative to neural networks. Theoretically, we derive new finite sample and asymptotic $L^2$ convergence rates for series estimators. We show that the rates can be improved in smooth settings by splitting the feature space and generating polynomial features separately for each partition. Empirically, we show that our proposed estimator, the BPR, can perform as well as more complex models with more parameters. Our estimator also performs close to state-of-the-art prediction methods in the benchmark MNIST handwritten digit dataset. We demonstrate that BPR performs as well as neural networks in crop classification using satellite data, a setting where prediction accuracy is critical and interpretability is often required for addressing research questions.