Mixed-Integer Linear Optimization for Cardinality-Constrained Random Forests

TL;DR

This paper introduces a mixed-integer linear optimization model for semi-supervised random forests, improving classification accuracy and correlation metrics in biased, limited-label scenarios.

Contribution

It develops a novel optimization-based approach for semi-supervised random forests, incorporating labeled and unlabeled data and class size information.

Findings

Improved accuracy over traditional random forests in biased samples

Better Matthews correlation coefficient with limited labeled data

Effective preprocessing and branching techniques for large problems

Abstract

Random forests are among the most famous algorithms for solving classification problems, in particular for large-scale data sets. Considering a set of labeled points and several decision trees, the method takes the majority vote to classify a new given point. In some scenarios, however, labels are only accessible for a proper subset of the given points. Moreover, this subset can be non-representative, e.g., due to collection bias. Semi-supervised learning considers the setting of labeled and unlabeled data and often improves the reliability of the results. In addition, it can be possible to obtain additional information about class sizes from undisclosed sources. We propose a mixed-integer linear optimization model for computing a semi-supervised random forest that covers the setting of labeled and unlabeled data points as well as the overall number of points in each class for a binary classification. Since the solution time rapidly grows as the number of variables increases, we present some problem-tailored preprocessing techniques and an intuitive branching rule. Our numerical results show that our approach leads to a better accuracy and a better Matthews correlation coefficient for biased samples compared to random forests by majority vote, even if only few labeled points are available.