prunAdag: an adaptive pruning-aware gradient method

TL;DR

This paper introduces prunAdag, an adaptive gradient method that classifies variables for targeted updates, enabling effective sparsification and demonstrating competitive convergence and performance in various applications.

Contribution

It extends previous relevance-based methods by incorporating a pruning-aware adaptive gradient approach with proven convergence guarantees.

Findings

Proves convergence with a global rate of O(\u001dlog(k)/\u001d extrac{1}{\u001d ext extrac{1}{",

Demonstrates competitive performance in multiple numerical experiments.

Enables a posteriori sparsification of model parameters.

Abstract

A pruning-aware adaptive gradient method is proposed which classifies the variables in two sets before updating them using different strategies. This technique extends the ``relevant/irrelevant" approach of Ding (2019) and Zimmer et al. (2022) and allows a posteriori sparsification of the solution of model parameter fitting problems. The new method is proved to be convergent with a global rate of decrease of the averaged gradient's norm of the form $\calO(\log(k)/\sqrt{k+1})$. Numerical experiments on several applications show that it is competitive.