Simplification of Forest Classifiers and Regressors

TL;DR

This paper introduces a method to simplify forest classifiers and regressors by sharing branching conditions while maintaining performance, using interval intersection algorithms and relaxations to balance simplicity and accuracy.

Contribution

It proposes a novel algorithm for sharing branching conditions in forests, with relaxations allowing controlled decision path changes, validated through extensive experiments.

Findings

Significant reduction in branching conditions without accuracy loss

Effective sharing achieved across multiple datasets and classifiers

Relaxation techniques improve the trade-off between simplicity and performance

Abstract

We study the problem of sharing as many branching conditions of a given forest classifier or regressor as possible while keeping classification performance. As a constraint for preventing from accuracy degradation, we first consider the one that the decision paths of all the given feature vectors must not change. For a branching condition that a value of a certain feature is at most a given threshold, the set of values satisfying such constraint can be represented as an interval. Thus, the problem is reduced to the problem of finding the minimum set intersecting all the constraint-satisfying intervals for each set of branching conditions on the same feature. We propose an algorithm for the original problem using an algorithm solving this problem efficiently. The constraint is relaxed later to promote further sharing of branching conditions by allowing decision path change of a certain ratio of the given feature vectors or allowing a certain number of non-intersected constraint-satisfying intervals. We also extended our algorithm for both the relaxations. The effectiveness of our method is demonstrated through comprehensive experiments using 21 datasets (13 classification and 8 regression datasets in UCI machine learning repository) and 4 classifiers/regressors (random forest, extremely randomized trees, AdaBoost and gradient boosting).