New Perspectives on $k$-Support and Cluster Norms

TL;DR

This paper introduces the box-norm, a new regularizer related to the k-support norm, extends it to matrices, and demonstrates its effectiveness in matrix completion and multitask learning.

Contribution

The paper defines the box-norm as a parameterized infimum of quadratics, relates it to the k-support norm, and extends these norms to matrices with spectral variants, improving algorithms and applications.

Findings

Spectral box-norm is equivalent to the cluster norm.

Centered spectral norms improve multitask learning.

Spectral k-support and box-norms achieve state-of-the-art results.

Abstract

We study a regularizer which is defined as a parameterized infimum of quadratics, and which we call the box-norm. We show that the k-support norm, a regularizer proposed by [Argyriou et al, 2012] for sparse vector prediction problems, belongs to this family, and the box-norm can be generated as a perturbation of the former. We derive an improved algorithm to compute the proximity operator of the squared box-norm, and we provide a method to compute the norm. We extend the norms to matrices, introducing the spectral k-support norm and spectral box-norm. We note that the spectral box-norm is essentially equivalent to the cluster norm, a multitask learning regularizer introduced by [Jacob et al. 2009a], and which in turn can be interpreted as a perturbation of the spectral k-support norm. Centering the norm is important for multitask learning and we also provide a method to use centered versions of the norms as regularizers. Numerical experiments indicate that the spectral k-support and box-norms and their centered variants provide state of the art performance in matrix completion and multitask learning problems respectively.