Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation

TL;DR

This paper introduces novel randomized algorithms for estimating specific matrix norms in a matrix-free context, leveraging matrix-vector multiplications, with applications in neural network regularization and adversarial attack mitigation.

Contribution

The paper presents new matrix-free algorithms for two-to-infinity and one-to-two norm estimation, including complexity bounds and practical applications in deep learning and recommender systems.

Findings

Algorithms effectively estimate matrix norms using only matrix-vector products.

Applications improve neural network regularization and robustness against adversarial attacks.

The methods demonstrate practical utility in real-world machine learning tasks.

Abstract

In this paper, we propose new randomized algorithms for estimating the two-to-infinity and one-to-two norms in a matrix-free setting, using only matrix-vector multiplications. Our methods are based on appropriate modifications of Hutchinson's diagonal estimator and its Hutch++ version. We provide oracle complexity bounds for both modifications. We further illustrate the practical utility of our algorithms for Jacobian-based regularization in deep neural network training on image classification tasks. We also demonstrate that our methodology can be applied to mitigate the effect of adversarial attacks in the domain of recommender systems.