Statistical Learnability of Generalized Additive Models based on Total Variation Regularization

TL;DR

This paper analyzes the statistical learnability of generalized additive models (GAMs) using total variation regularization, providing tight bounds on their complexity and generalization error in classification tasks.

Contribution

It introduces a TV-based complexity measure for GAMs and derives tight Rademacher complexity bounds and generalization error estimates.

Findings

GAMs with TV regularization have Rademacher complexity of O(√(log p / m)).

The bounds are tight in terms of sample size and number of variables.

Provides finite-sample generalization error bounds for classification.

Abstract

A generalized additive model (GAM, Hastie and Tibshirani (1987)) is a nonparametric model by the sum of univariate functions with respect to each explanatory variable, i.e., $f({\mathbf x}) = \sum f_j(x_j)$, where $x_j\in\mathbb{R}$ is $j$-th component of a sample ${\mathbf x}\in \mathbb{R}^p$. In this paper, we introduce the total variation (TV) of a function as a measure of the complexity of functions in $L^1_{\rm c}(\mathbb{R})$-space. Our analysis shows that a GAM based on TV-regularization exhibits a Rademacher complexity of $O(\sqrt{\frac{\log p}{m}})$, which is tight in terms of both $m$ and $p$ in the agnostic case of the classification problem. In result, we obtain generalization error bounds for finite samples according to work by Bartlett and Mandelson (2002).