Numerical evaluation of dual norms via the MM algorithm

TL;DR

This paper introduces an efficient algorithm based on the MM principle for numerically computing dual norms of sparsity-inducing regularizations, which are crucial in optimization and statistical learning but often lack analytical solutions.

Contribution

The paper proposes a novel MM algorithm that reparametrizes the dual norm computation as an unconstrained optimization problem with a barrier, enabling efficient calculation for complex norms.

Findings

Algorithm accurately computes dual norms in high dimensions.

Method outperforms existing approaches in speed and accuracy.

Validated through extensive simulations.

Abstract

We deal with the problem of numerically computing the dual norm, which is important to study sparsity-inducing regularizations (Jenatton et al. 2011,Bach et al. 2012). The dual norms find application in optimization and statistical learning, for example, in the design of working-set strategies, for characterizing dual gradient methods, for dual decompositions and in the definition of augmented Lagrangian functions. Nevertheless, the dual norm of some well-known sparsity-inducing regolarization methods are not analytically available. Examples are the overlap group $\ell_2$-norm of (Jenatton et al. 2011) and the elastic net norm of Zhou and Hastie (2005). Therefore we resort to the Majorization-Minimization principle of Lange (2016) to provide an efficient algorithm that leverages a reparametrization of the dual constrained optimization problem as unconstrained optimization with barrier. Extensive simulation experiments have been performed in order to verify the correctness of operation, and evaluate the performance of the proposed method. Our results demonstrate the effectiveness of the algorithm in retrieving the dual norm even for large dimensions.