Diversity and degrees of freedom in regression ensembles

TL;DR

This paper explores how diversity in regression ensembles influences their capacity, linking it to degrees of freedom, and demonstrates how to optimize diversity for improved performance, including in deep neural networks.

Contribution

It provides an exact formula for degrees of freedom in NCL ensembles, connecting diversity to regularisation and offering a method to tune the diversity parameter effectively.

Findings

Diversity acts as inverse regularisation in ensembles.

Optimal diversity improves performance in noisy settings.

Extension of the theory to deep neural network ensembles.

Abstract

Ensemble methods are a cornerstone of modern machine learning. The performance of an ensemble depends crucially upon the level of diversity between its constituent learners. This paper establishes a connection between diversity and degrees of freedom (i.e. the capacity of the model), showing that diversity may be viewed as a form of inverse regularisation. This is achieved by focusing on a previously published algorithm Negative Correlation Learning (NCL), in which model diversity is explicitly encouraged through a diversity penalty term in the loss function. We provide an exact formula for the effective degrees of freedom in an NCL ensemble with fixed basis functions, showing that it is a continuous, convex and monotonically increasing function of the diversity parameter. We demonstrate a connection to Tikhonov regularisation and show that, with an appropriately chosen diversity parameter, an NCL ensemble can always outperform the unregularised ensemble in the presence of noise. We demonstrate the practical utility of our approach by deriving a method to efficiently tune the diversity parameter. Finally, we use a Monte-Carlo estimator to extend the connection between diversity and degrees of freedom to ensembles of deep neural networks.