Exploiting random projections and sparsity with random forests and gradient boosting methods -- Application to multi-label and multi-output learning, random forest model compression and leveraging input sparsity

TL;DR

This paper introduces novel methods to enhance decision tree-based models like random forests and gradient boosting for high-dimensional, sparse, and large-scale data, improving scalability and efficiency in multi-label and multi-output tasks.

Contribution

It proposes original algorithms to improve decision tree methods for high-dimensional outputs, large datasets with memory constraints, and sparse input spaces, advancing scalable supervised learning.

Findings

Enhanced model performance on high-dimensional output spaces

Reduced memory usage for large datasets during prediction

Improved handling of input sparsity in decision tree models

Abstract

Within machine learning, the supervised learning field aims at modeling the input-output relationship of a system, from past observations of its behavior. Decision trees characterize the input-output relationship through a series of nested $if-then-else$ questions, the testing nodes, leading to a set of predictions, the leaf nodes. Several of such trees are often combined together for state-of-the-art performance: random forest ensembles average the predictions of randomized decision trees trained independently in parallel, while tree boosting ensembles train decision trees sequentially to refine the predictions made by the previous ones. The emergence of new applications requires scalable supervised learning algorithms in terms of computational power and memory space with respect to the number of inputs, outputs, and observations without sacrificing accuracy. In this thesis, we identify three main areas where decision tree methods could be improved for which we provide and evaluate original algorithmic solutions: (i) learning over high dimensional output spaces, (ii) learning with large sample datasets and stringent memory constraints at prediction time and (iii) learning over high dimensional sparse input spaces.