Multi-level projection with exponential parallel speedup; Application to sparse auto-encoders neural networks

TL;DR

This paper introduces a novel multi-level projection method that significantly accelerates structured projections like the $\, ext{l}_{1, ext{infinity}}$ norm, achieving exponential parallel speedup and improving efficiency in neural network applications.

Contribution

The paper presents a bi-level and multi-level projection approach that reduces computational complexity and achieves exponential parallel speedup for structured norms, extending to tensors.

Findings

Projection is twice as fast as existing Euclidean algorithms.

Method maintains accuracy while enhancing sparsity in neural networks.

Achieves linear parallel speedup up to an exponential factor.

Abstract

The $\ell_{1,\infty}$ norm is an efficient structured projection but the complexity of the best algorithm is unfortunately $\mathcal{O}\big(n m \log(n m)\big)$ for a matrix in $\mathbb{R}^{n\times m}$. In this paper, we propose a new bi-level projection method for which we show that the time complexity for the $\ell_{1,\infty}$ norm is only $\mathcal{O}\big(n m \big)$ for a matrix in $\mathbb{R}^{n\times m}$, and $\mathcal{O}\big(n + m \big)$ with full parallel power. We generalize our method to tensors and we propose a new multi-level projection, having an induced decomposition that yields a linear parallel speedup up to an exponential speedup factor, resulting in a time complexity lower-bounded by the sum of the dimensions, instead of the product of the dimensions. we provide a large base of implementation of our framework for bi-level and tri-level (matrices and tensors) for various norms and provides also the parallel implementation. Experiments show that our projection is $2$ times faster than the actual fastest Euclidean algorithms while providing same accuracy and better sparsity in neural networks applications.