A direct approach to tree-guided feature aggregation for high-dimensional regression

TL;DR

This paper introduces a novel tree-guided regularization method for high-dimensional regression that efficiently estimates and aggregates features without overparameterization, supported by theoretical analysis and empirical validation.

Contribution

It proposes a new regularization scheme with a non-iterative algorithm for hierarchical feature aggregation, improving over existing methods in speed and accuracy.

Findings

The method achieves faster or comparable error rates to existing approaches.

The proximal operator can be computed with a single pass, non-iterative algorithm.

Simulation studies confirm the theoretical advantages and practical effectiveness.

Abstract

In high-dimensional linear models, sparsity is often exploited to reduce variability and achieve parsimony. Equi-sparsity, where one assumes that predictors can be aggregated into groups sharing the same effects, is an alternative parsimonious structure that can be more suitable in certain applications. Previous work has clearly demonstrated the benefits of exploiting equi-sparsity in the presence of ``rare features'' (Yan and Bien 2021). In this work, we propose a new tree-guided regularization scheme for simultaneous estimation and feature aggregation. Unlike existing methods, our estimator avoids synthetic overparameterization and its detrimental effects. Even though our penalty is applied to hierarchically overlapped groups, we show that its proximal operator can be solved with a one-pass, non-iterative algorithm. Novel techniques are developed to study the finite-sample error bound of this seminorm-induced regularizer under least squares and binomial deviance losses. Theoretically, compared to existing methods, the proposed method offers a faster or equivalent rate depending on the true equi-sparisty structure. Extensive simulation studies verify these findings. Finally, we illustrate the usefulness of the proposed method with an application to a microbiome dataset, where we conduct post-selection inference on the aggregated features' effects.