Boosting-Based Sequential Meta-Tree Ensemble Construction for Improved Decision Trees

TL;DR

This paper introduces a boosting-based method to construct ensembles of meta-trees, which are designed to prevent overfitting and improve predictive performance beyond traditional decision tree ensembles.

Contribution

It proposes a novel boosting approach to build multiple meta-trees, addressing overfitting issues and enhancing ensemble effectiveness compared to conventional decision tree ensembles.

Findings

Ensembles of meta-trees outperform single meta-trees and traditional decision tree ensembles.

Meta-tree ensembles effectively prevent overfitting caused by deep trees.

Experimental results show improved accuracy on synthetic and benchmark datasets.

Abstract

A decision tree is one of the most popular approaches in machine learning fields. However, it suffers from the problem of overfitting caused by overly deepened trees. Then, a meta-tree is recently proposed. It solves the problem of overfitting caused by overly deepened trees. Moreover, the meta-tree guarantees statistical optimality based on Bayes decision theory. Therefore, the meta-tree is expected to perform better than the decision tree. In contrast to a single decision tree, it is known that ensembles of decision trees, which are typically constructed boosting algorithms, are more effective in improving predictive performance. Thus, it is expected that ensembles of meta-trees are more effective in improving predictive performance than a single meta-tree, and there are no previous studies that construct multiple meta-trees in boosting. Therefore, in this study, we propose a method to construct multiple meta-trees using a boosting approach. Through experiments with synthetic and benchmark datasets, we conduct a performance comparison between the proposed methods and the conventional methods using ensembles of decision trees. Furthermore, while ensembles of decision trees can cause overfitting as well as a single decision tree, experiments confirmed that ensembles of meta-trees can prevent overfitting due to the tree depth.