Hierarchical Total Variations and Doubly Penalized ANOVA Modeling for Multivariate Nonparametric Regression

TL;DR

This paper introduces a novel doubly penalized ANOVA modeling approach for multivariate nonparametric regression, utilizing hierarchical total variations and specialized basis functions to achieve sparse, interpretable component functions with improved predictive performance.

Contribution

The paper develops a new hierarchical total variation framework and basis functions for doubly penalized ANOVA, enabling sparse, interpretable multivariate nonparametric models with efficient estimation algorithms.

Findings

Outperforms existing methods like MARS, boosting, and random forests in accuracy.

Produces simpler, more interpretable component functions.

Demonstrates effectiveness through extensive simulations and real data applications.

Abstract

For multivariate nonparametric regression, functional analysis-of-variance (ANOVA) modeling aims to capture the relationship between a response and covariates by decomposing the unknown function into various components, representing main effects, two-way interactions, etc. Such an approach has been pursued explicitly in smoothing spline ANOVA modeling and implicitly in various greedy methods such as MARS. We develop a new method for functional ANOVA modeling, based on doubly penalized estimation using total-variation and empirical-norm penalties, to achieve sparse selection of component functions and their knots. For this purpose, we formulate a new class of hierarchical total variations, which measures total variations at different levels including main effects and multi-way interactions, possibly after some order of differentiation. Furthermore, we derive suitable basis functions for multivariate splines such that the hierarchical total variation can be represented as a regular Lasso penalty, and hence we extend a previous backfitting algorithm to handle doubly penalized estimation for ANOVA modeling. We present extensive numerical experiments on simulations and real data to compare our method with existing methods including MARS, tree boosting, and random forest. The results are very encouraging and demonstrate considerable gains from our method in both prediction or classification accuracy and simplicity of the fitted functions.