Sparse Estimation by Exponential Weighting

TL;DR

This paper introduces a new sparse estimation method using exponential weights that effectively exploits sparsity in regression models, providing strong theoretical guarantees and efficient implementation without restrictive conditions.

Contribution

It develops a sparsity pattern aggregation approach with oracle inequalities applicable across various sparsity frameworks, and offers an efficient algorithm for practical use.

Findings

Theoretical sparsity oracle inequalities hold without dictionary conditions.

The proposed method performs favorably in numerical experiments.

Applicable to ordinary, fused, and group sparsity models.

Abstract

Consider a regression model with fixed design and Gaussian noise where the regression function can potentially be well approximated by a function that admits a sparse representation in a given dictionary. This paper resorts to exponential weights to exploit this underlying sparsity by implementing the principle of sparsity pattern aggregation. This model selection take on sparse estimation allows us to derive sparsity oracle inequalities in several popular frameworks, including ordinary sparsity, fused sparsity and group sparsity. One striking aspect of these theoretical results is that they hold under no condition in the dictionary. Moreover, we describe an efficient implementation of the sparsity pattern aggregation principle that compares favorably to state-of-the-art procedures on some basic numerical examples.