Weighted Sum-of-Trees Model for Clustered Data

TL;DR

This paper introduces a weighted sum-of-trees model for clustered data that learns individual group-specific trees and combines them with learned weights, improving prediction accuracy and interpretability over traditional models.

Contribution

The paper presents a novel sum-of-trees approach that models each group separately and uses learned weights for better out-of-sample predictions and group similarity inference.

Findings

Outperforms traditional decision trees and random forests in simulations

Enables inference on group similarities through tree structures and variable importances

Effective on real-world sarcoma data from The Cancer Genome Atlas

Abstract

Clustered data, which arise when observations are nested within groups, are incredibly common in clinical, education, and social science research. Traditionally, a linear mixed model, which includes random effects to account for within-group correlation, would be used to model the observed data and make new predictions on unseen data. Some work has been done to extend the mixed model approach beyond linear regression into more complex and non-parametric models, such as decision trees and random forests. However, existing methods are limited to using the global fixed effects for prediction on data from out-of-sample groups, effectively assuming that all clusters share a common outcome model. We propose a lightweight sum-of-trees model in which we learn a decision tree for each sample group. We combine the predictions from these trees using weights so that out-of-sample group predictions are more closely aligned with the most similar groups in the training data. This strategy also allows for inference on the similarity across groups in the outcome prediction model, as the unique tree structures and variable importances for each group can be directly compared. We show our model outperforms traditional decision trees and random forests in a variety of simulation settings. Finally, we showcase our method on real-world data from the sarcoma cohort of The Cancer Genome Atlas, where patient samples are grouped by sarcoma subtype.