Analyze Additive and Interaction Effects via Collaborative Trees

TL;DR

This paper introduces Collaborative Trees, a new regression tree model that analyzes additive and interaction effects of features, providing visual and numerical tools, with theoretical insights into its performance and stability.

Contribution

The paper presents a novel Collaborative Trees model with regularization, enabling detailed analysis of feature effects and interactions, supported by theoretical and empirical validation.

Findings

Collaborative Trees effectively decompose feature effects.

Network diagrams visually depict additive and interaction effects.

Theoretical analysis links the model to matching pursuit, explaining its effectiveness.

Abstract

We present Collaborative Trees, a novel tree model designed for regression prediction, along with its bagging version, which aims to analyze complex statistical associations between features and uncover potential patterns inherent in the data. We decompose the mean decrease in impurity from the proposed tree model to analyze the additive and interaction effects of features on the response variable. Additionally, we introduce network diagrams to visually depict how each feature contributes additively to the response and how pairs of features contribute interaction effects. Through a detailed demonstration using an embryo growth dataset, we illustrate how the new statistical tools aid data analysis, both visually and numerically. Moreover, we delve into critical aspects of tree modeling, such as prediction performance, inference stability, and bias in feature importance measures, leveraging real datasets and simulation experiments for comprehensive discussions. On the theory side, we show that Collaborative Trees, built upon a ``sum of trees'' approach with our own innovative tree model regularization, exhibit characteristics akin to matching pursuit, under the assumption of high-dimensional independent binary input features (or one-hot feature groups). This newfound link sheds light on the superior capability of our tree model in estimating additive effects of features, a crucial factor for accurate interaction effect estimation.