SMART: A Flexible Approach to Regression using Spline-Based Multivariate Adaptive Regression Trees

TL;DR

SMART combines decision trees and MARS to effectively model both discontinuities and continuous relationships in data, improving predictive accuracy over existing methods.

Contribution

Introduces SMART, a novel method that integrates decision trees with MARS to better handle discontinuities and complex relationships in regression tasks.

Findings

SMART outperforms state-of-the-art methods on various datasets.

The method effectively captures both discontinuities and smooth relationships.

Open-source implementation available for practitioners.

Abstract

Decision trees are powerful for predictive modeling but often suffer from high variance when modeling continuous relationships. While algorithms like Multivariate Adaptive Regression Splines (MARS) excel at capturing such continuous relationships, they perform poorly when modeling discontinuities. To address the limitations of both approaches, we introduce Spline-based Multivariate Adaptive Regression Trees (SMART), which uses a decision tree to identify subsets of data with distinct continuous relationships and then leverages MARS to fit these relationships independently. Unlike other methods that rely on the tree structure to model interaction and higher-order terms, SMART leverages MARS's native ability to handle these terms, allowing the tree to focus solely on identifying discontinuities in the relationship. We test SMART on various datasets, demonstrating its improvement over state-of-the-art methods in such cases. Additionally, we provide an open-source implementation of our method to be used by practitioners.