Canonical Correlation Forests

TL;DR

Canonical correlation forests (CCFs) are a novel ensemble method using hyperplane splits based on canonical correlations, offering improved accuracy and efficiency over traditional random forests, especially with correlated inputs.

Contribution

This paper introduces CCFs, a new decision tree ensemble method utilizing canonical correlation-based splits and projection bootstrapping, enhancing performance and robustness over existing methods.

Findings

CCFs outperform random forests in accuracy and speed.

CCFs handle correlated inputs more naturally.

CCFs outperform 179 classifiers in a recent survey.

Abstract

We introduce canonical correlation forests (CCFs), a new decision tree ensemble method for classification and regression. Individual canonical correlation trees are binary decision trees with hyperplane splits based on local canonical correlation coefficients calculated during training. Unlike axis-aligned alternatives, the decision surfaces of CCFs are not restricted to the coordinate system of the inputs features and therefore more naturally represent data with correlated inputs. CCFs naturally accommodate multiple outputs, provide a similar computational complexity to random forests, and inherit their impressive robustness to the choice of input parameters. As part of the CCF training algorithm, we also introduce projection bootstrapping, a novel alternative to bagging for oblique decision tree ensembles which maintains use of the full dataset in selecting split points, often leading to improvements in predictive accuracy. Our experiments show that, even without parameter tuning, CCFs out-perform axis-aligned random forests and other state-of-the-art tree ensemble methods on both classification and regression problems, delivering both improved predictive accuracy and faster training times. We further show that they outperform all of the 179 classifiers considered in a recent extensive survey.