Sparsifying sums of norms

TL;DR

This paper presents a method to approximate sums of multiple norms with a sparse weighted combination, maintaining accuracy while significantly reducing the number of terms, with efficient algorithms for certain classes of norms.

Contribution

The authors introduce a sparsification technique for sums of norms, providing bounds on the number of non-zero weights and efficient algorithms for finding these weights in specific cases.

Findings

Achieves approximation within epsilon with only O(epsilon^{-2} n log(n/epsilon) (log n)^{2.5}) non-zero weights.

Provides a high-probability algorithm to compute weights in time depending on the number of norms and evaluation time.

Extends to sums of pth powers of norms under p-uniform smoothness conditions.

Abstract

For any norms $N_1,\ldots,N_m$ on $\mathbb{R}^n$ and $N(x) := N_1(x)+\cdots+N_m(x)$, we show there is a sparsified norm $\tilde{N}(x) = w_1 N_1(x) + \cdots + w_m N_m(x)$ such that $|N(x) - \tilde{N}(x)| \leq \epsilon N(x)$ for all $x \in \mathbb{R}^n$, where $w_1,\ldots,w_m$ are non-negative weights, of which only $O(\epsilon^{-2} n \log(n/\epsilon) (\log n)^{2.5} )$ are non-zero. Additionally, if $N$ is $\mathrm{poly}(n)$-equivalent to the Euclidean norm on $\mathbb{R}^n$, then such weights can be found with high probability in time $O(m (\log n)^{O(1)} + \mathrm{poly}(n)) T$, where $T$ is the time required to evaluate a norm $N_i$. This immediately yields analogous statements for sparsifying sums of symmetric submodular functions. More generally, we show how to sparsify sums of $p$th powers of norms when the sum is $p$-uniformly smooth.