Evaluation of Bagging Predictors with Kernel Density Estimation and Bagging Score

TL;DR

This paper introduces a novel method using Kernel Density Estimation with Neural Networks to improve ensemble predictions in bagging, providing a confidence score and outperforming traditional averaging methods.

Contribution

The paper presents a new approach that determines a representative prediction and confidence measure for bagging ensembles using KDE and neural networks, without optimization or feature selection.

Findings

The new method yields more accurate predictions than mean or median aggregation.

It provides a confidence score (Bagging Score) reflecting prediction reliability.

The approach ranks highly compared to existing nonlinear regression methods.

Abstract

For a larger set of predictions of several differently trained machine learning models, known as bagging predictors, the mean of all predictions is taken by default. Nevertheless, this proceeding can deviate from the actual ground truth in certain parameter regions. An approach is presented to determine a representative y_BS from such a set of predictions using Kernel Density Estimation (KDE) in nonlinear regression with Neural Networks (NN) which simultaneously provides an associated quality criterion beta_BS, called Bagging Score (BS), that reflects the confidence of the obtained ensemble prediction. It is shown that working with the new approach better predictions can be made than working with the common use of mean or median. In addition to this, the used method is contrasted to several approaches of nonlinear regression from the literatur, resulting in a top ranking in each of the calculated error values without using any optimization or feature selection technique.