On Subagging Boosted Probit Model Trees

TL;DR

This paper introduces SBPMT, a hybrid bagging-boosting classification algorithm using Probit Model Trees, with theoretical guarantees and practical advantages demonstrated through simulations and comparisons.

Contribution

The paper proposes SBPMT, a novel hybrid bagging-boosting algorithm with a new Probit Model Tree base classifier, and provides theoretical analysis and empirical validation of its effectiveness.

Findings

SBPMT is consistent under certain conditions.

Increasing subagging reduces generalization error.

More ProbitBoost iterations improve performance with fewer boosting steps.

Abstract

With the insight of variance-bias decomposition, we design a new hybrid bagging-boosting algorithm named SBPMT for classification problems. For the boosting part of SBPMT, we propose a new tree model called Probit Model Tree (PMT) as base classifiers in AdaBoost procedure. For the bagging part, instead of subsampling from the dataset at each step of boosting, we perform boosted PMTs on each subagged dataset and combine them into a powerful "committee", which can be viewed an incomplete U-statistic. Our theoretical analysis shows that (1) SBPMT is consistent under certain assumptions, (2) Increase the subagging times can reduce the generalization error of SBPMT to some extent and (3) Large number of ProbitBoost iterations in PMT can benefit the performance of SBPMT with fewer steps in the AdaBoost part. Those three properties are verified by a famous simulation designed by Mease and Wyner (2008). The last two points also provide a useful guidance in model tuning. A comparison of performance with other state-of-the-art classification methods illustrates that the proposed SBPMT algorithm has competitive prediction power in general and performs significantly better in some cases.