A hybrid ensemble method with negative correlation learning for regression

TL;DR

This paper introduces a hybrid ensemble regression method that automatically selects and weights diverse sub-models using negative correlation learning, resulting in improved performance over traditional averaging and weighting techniques.

Contribution

It proposes an innovative optimization approach incorporating negative correlation learning for automatic sub-model selection and weighting in ensemble regression.

Findings

Ensemble performance surpasses simple averaging and existing weighting methods.

The method achieves comparable accuracy to the optimal sub-models.

Inclusion of regularization enhances ensemble performance.

Abstract

Hybrid ensemble, an essential branch of ensembles, has flourished in the regression field, with studies confirming diversity's importance. However, previous ensembles consider diversity in the sub-model training stage, with limited improvement compared to single models. In contrast, this study automatically selects and weights sub-models from a heterogeneous model pool. It solves an optimization problem using an interior-point filtering linear-search algorithm. The objective function innovatively incorporates negative correlation learning as a penalty term, with which a diverse model subset can be selected. The best sub-models from each model class are selected to build the NCL ensemble, which performance is better than the simple average and other state-of-the-art weighting methods. It is also possible to improve the NCL ensemble with a regularization term in the objective function. In practice, it is difficult to conclude the optimal sub-model for a dataset prior due to the model uncertainty. Regardless, our method would achieve comparable accuracy as the potential optimal sub-models. In conclusion, the value of this study lies in its ease of use and effectiveness, allowing the hybrid ensemble to embrace diversity and accuracy.