To Boost or not to Boost: On the Limits of Boosted Neural Networks

TL;DR

This paper investigates the effectiveness of boosting neural networks versus decision trees, revealing that boosting improves decision tree performance but not neural networks, where single models often outperform ensembles.

Contribution

It provides theoretical insights into the representational differences between boosted decision trees and neural networks, supported by experimental validation on object recognition datasets.

Findings

Boosted decision trees generalize better than single trees with same parameters.

Boosted neural networks do not outperform single neural networks with equivalent total parameters.

Neural networks benefit less from boosting compared to decision trees.

Abstract

Boosting is a method for finding a highly accurate hypothesis by linearly combining many ``weak" hypotheses, each of which may be only moderately accurate. Thus, boosting is a method for learning an ensemble of classifiers. While boosting has been shown to be very effective for decision trees, its impact on neural networks has not been extensively studied. We prove one important difference between sums of decision trees compared to sums of convolutional neural networks (CNNs) which is that a sum of decision trees cannot be represented by a single decision tree with the same number of parameters while a sum of CNNs can be represented by a single CNN. Next, using standard object recognition datasets, we verify experimentally the well-known result that a boosted ensemble of decision trees usually generalizes much better on testing data than a single decision tree with the same number of parameters. In contrast, using the same datasets and boosting algorithms, our experiments show the opposite to be true when using neural networks (both CNNs and multilayer perceptrons (MLPs)). We find that a single neural network usually generalizes better than a boosted ensemble of smaller neural networks with the same total number of parameters.