The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest

TL;DR

This paper provides the first theoretical analysis of deep forests, showing how depth, width, and tree size influence their expressiveness, with depth playing a particularly significant role.

Contribution

It offers the first upper and lower bounds on deep forest approximation complexity related to key hyperparameters, highlighting depth's exponential impact.

Findings

Depth exponentially enhances expressiveness.

Width and tree size have comparatively lesser effects.

Experimental results validate theoretical bounds.

Abstract

Random forests are classical ensemble algorithms that construct multiple randomized decision trees and aggregate their predictions using naive averaging. \citet{zhou2019deep} further propose a deep forest algorithm with multi-layer forests, which outperforms random forests in various tasks. The performance of deep forests is related to three hyperparameters in practice: depth, width, and tree size, but little has been known about its theoretical explanation. This work provides the first upper and lower bounds on the approximation complexity of deep forests concerning the three hyperparameters. Our results confirm the distinctive role of depth, which can exponentially enhance the expressiveness of deep forests compared with width and tree size. Experiments confirm the theoretical findings.