Improve Deep Forest with Learnable Layerwise Augmentation Policy Schedule

TL;DR

This paper enhances Deep Forest by introducing learnable, layer-specific data augmentation policies and checkpoint ensembles, significantly improving performance on tabular classification tasks and setting new state-of-the-art results.

Contribution

It proposes a novel learnable augmentation schedule and a population-based search algorithm to optimize Deep Forest's layerwise data augmentation, reducing overfitting and boosting accuracy.

Findings

Achieved new state-of-the-art results on tabular classification benchmarks.

Demonstrated effective transferability of learned policies to other Deep Forest variants.

Outperformed traditional shallow and deep models, including neural networks and AutoML methods.

Abstract

As a modern ensemble technique, Deep Forest (DF) employs a cascading structure to construct deep models, providing stronger representational power compared to traditional decision forests. However, its greedy multi-layer learning procedure is prone to overfitting, limiting model effectiveness and generalizability. This paper presents an optimized Deep Forest, featuring learnable, layerwise data augmentation policy schedules. Specifically, We introduce the Cut Mix for Tabular data (CMT) augmentation technique to mitigate overfitting and develop a population-based search algorithm to tailor augmentation intensity for each layer. Additionally, we propose to incorporate outputs from intermediate layers into a checkpoint ensemble for more stable performance. Experimental results show that our method sets new state-of-the-art (SOTA) benchmarks in various tabular classification tasks, outperforming shallow tree ensembles, deep forests, deep neural network, and AutoML competitors. The learned policies also transfer effectively to Deep Forest variants, underscoring its potential for enhancing non-differentiable deep learning modules in tabular signal processing.