On Deep Ensemble Learning from a Function Approximation Perspective

TL;DR

This paper introduces a deep ensemble learning framework that combines multiple models to universally approximate functions, with theoretical bounds on the number of models needed depending on the ensemble depth and input dimension.

Contribution

It provides a theoretical foundation for deep ensemble learning, establishing bounds on the number of models required for universal approximation based on ensemble depth and input dimension.

Findings

Deep ensemble can achieve universal approximation of functions.

Number of models needed is 2^d for single-layer ensembles.

Deeper ensembles exponentially reduce the number of models required.

Abstract

In this paper, we propose to provide a general ensemble learning framework based on deep learning models. Given a group of unit models, the proposed deep ensemble learning framework will effectively combine their learning results via a multilayered ensemble model. In the case when the unit model mathematical mappings are bounded, sigmoidal and discriminatory, we demonstrate that the deep ensemble learning framework can achieve a universal approximation of any functions from the input space to the output space. Meanwhile, to achieve such a performance, the deep ensemble learning framework also impose a strict constraint on the number of involved unit models. According to the theoretic proof provided in this paper, given the input feature space of dimension d, the required unit model number will be 2d, if the ensemble model involves one single layer. Furthermore, as the ensemble component goes deeper, the number of required unit model is proved to be lowered down exponentially.