MP-Boost: Minipatch Boosting via Adaptive Feature and Observation Sampling

TL;DR

MP-Boost is a fast, interpretable boosting algorithm that adaptively selects small data subsets and features, achieving accuracy comparable to AdaBoost and gradient boosting while improving computational efficiency.

Contribution

This paper introduces MP-Boost, a novel boosting method that adaptively learns to select minipatches of data and features, enhancing speed and interpretability over traditional boosting algorithms.

Findings

MP-Boost achieves comparable accuracy to AdaBoost and gradient boosting.

The method is computationally faster due to small data subset learning.

MP-Boost provides improved interpretability through learned probability distributions.

Abstract

Boosting methods are among the best general-purpose and off-the-shelf machine learning approaches, gaining widespread popularity. In this paper, we seek to develop a boosting method that yields comparable accuracy to popular AdaBoost and gradient boosting methods, yet is faster computationally and whose solution is more interpretable. We achieve this by developing MP-Boost, an algorithm loosely based on AdaBoost that learns by adaptively selecting small subsets of instances and features, or what we term minipatches (MP), at each iteration. By sequentially learning on tiny subsets of the data, our approach is computationally faster than other classic boosting algorithms. Also as it progresses, MP-Boost adaptively learns a probability distribution on the features and instances that upweight the most important features and challenging instances, hence adaptively selecting the most relevant minipatches for learning. These learned probability distributions also aid in interpretation of our method. We empirically demonstrate the interpretability, comparative accuracy, and computational time of our approach on a variety of binary classification tasks.