Neural Random Forest Imitation

TL;DR

This paper introduces Neural Random Forest Imitation, a method that converts random forests into efficient neural networks through imitation learning, resulting in models that are more parameter-efficient and often more accurate, especially with limited data.

Contribution

The paper proposes a novel imitation learning approach to transform random forests into neural networks, improving efficiency and performance over existing direct mapping methods.

Findings

Neural networks learned via imitation outperform direct mappings in efficiency.

The approach achieves comparable or better accuracy with fewer parameters.

Models perform well even with very limited training data.

Abstract

We present Neural Random Forest Imitation - a novel approach for transforming random forests into neural networks. Existing methods propose a direct mapping and produce very inefficient architectures. In this work, we introduce an imitation learning approach by generating training data from a random forest and learning a neural network that imitates its behavior. This implicit transformation creates very efficient neural networks that learn the decision boundaries of a random forest. The generated model is differentiable, can be used as a warm start for fine-tuning, and enables end-to-end optimization. Experiments on several real-world benchmark datasets demonstrate superior performance, especially when training with very few training examples. Compared to state-of-the-art methods, we significantly reduce the number of network parameters while achieving the same or even improved accuracy due to better generalization.