A Framework for an Assessment of the Kernel-target Alignment in Tree Ensemble Kernel Learning

TL;DR

This paper investigates the kernel-target alignment in tree ensemble kernel learning, demonstrating that strong alignment correlates with good performance and that relevant information is concentrated in target-aligned components, supported by simulations and real data.

Contribution

It introduces an eigenanalysis-based framework to assess kernel-target alignment in tree ensemble kernels, filling a gap in understanding their effectiveness.

Findings

Strong kernel-target alignment correlates with high performance.

Relevant information is concentrated in low-dimensional, target-aligned components.

Tree ensemble kernels maintain persistent target-aligned components across data sets.

Abstract

Kernels ensuing from tree ensembles such as random forest (RF) or gradient boosted trees (GBT), when used for kernel learning, have been shown to be competitive to their respective tree ensembles (particularly in higher dimensional scenarios). On the other hand, it has been also shown that performance of the kernel algorithms depends on the degree of the kernel-target alignment. However, the kernel-target alignment for kernel learning based on the tree ensembles has not been investigated and filling this gap is the main goal of our work. Using the eigenanalysis of the kernel matrix, we demonstrate that for continuous targets good performance of the tree-based kernel learning is associated with strong kernel-target alignment. Moreover, we show that well performing tree ensemble based kernels are characterized by strong target aligned components that are expressed through scalar products between the eigenvectors of the kernel matrix and the target. This suggests that when tree ensemble based kernel learning is successful, relevant information for the supervised problem is concentrated near lower dimensional manifold spanned by the target aligned components. Persistence of the strong target aligned components in tree ensemble based kernels is further supported by sensitivity analysis via landmark learning. In addition to a comprehensive simulation study, we also provide experimental results from several real life data sets that are in line with the simulations.