Vectorized Adaptive Histograms for Sparse Oblique Forests

TL;DR

This paper introduces a method to optimize the training of sparse oblique random forests by dynamically switching between histogram and sorting splits and leveraging vector intrinsics, significantly speeding up training.

Contribution

It proposes a novel approach for adaptive histogram construction and implementation optimizations for sparse oblique forests, improving training efficiency.

Findings

Speedup of 1.7-2.5x over existing oblique forests

Speedup of 1.5-2x over standard random forests

Effective GPU and hybrid CPU-GPU implementations

Abstract

Classification using sparse oblique random forests provides guarantees on uncertainty and confidence while controlling for specific error types. However, they use more data and more compute than other tree ensembles because they create deep trees and need to sort or histogram linear combinations of data at runtime. We provide a method for dynamically switching between histograms and sorting to find the best split. We further optimize histogram construction using vector intrinsics. Evaluating this on large datasets, our optimizations speedup training by 1.7-2.5x compared to existing oblique forests and 1.5-2x compared to standard random forests. We also provide a GPU and hybrid CPU-GPU implementation.