Towards a Systematic Approach to Design New Ensemble Learning Algorithms

TL;DR

This paper introduces a systematic framework for designing new ensemble learning algorithms by applying a modern bias-variance-diversity decomposition theory, leading to the creation of 21 novel algorithms with improved performance.

Contribution

It develops a structured approach to create ensemble algorithms based on a unified diversity theory, resulting in 21 new algorithms that outperform existing methods.

Findings

Most new algorithms using snapshot strategy outperform baselines

The approach provides a clear pathway for future ensemble algorithm development

New algorithms show superior predictive performance across datasets

Abstract

Ensemble learning has been a focal point of machine learning research due to its potential to improve predictive performance. This study revisits the foundational work on ensemble error decomposition, historically confined to bias-variance-covariance analysis for regression problems since the 1990s. Recent advancements introduced a "unified theory of diversity," which proposes an innovative bias-variance-diversity decomposition framework. Leveraging this contemporary understanding, our research systematically explores the application of this decomposition to guide the creation of new ensemble learning algorithms. Focusing on regression tasks, we employ neural networks as base learners to investigate the practical implications of this theoretical framework. This approach used 7 simple ensemble methods, we name them strategies, for neural networks that were used to generate 21 new ensemble algorithms. Among these, most of the methods aggregated with the snapshot strategy, one of the 7 strategies used, showcase superior predictive performance across diverse datasets w.r.t. the Friedman rank test with the Conover post-hoc test. Our systematic design approach contributes a suite of effective new algorithms and establishes a structured pathway for future ensemble learning algorithm development.