Permutation Randomization on Nonsmooth Nonconvex Optimization: A Theoretical and Experimental Study

TL;DR

This paper investigates how permutation randomization influences gradient-based optimization in nonsmooth nonconvex problems, providing both theoretical insights and empirical evidence of its benefits in convergence and practical performance.

Contribution

It offers the first theoretical analysis of permutation randomization's role in nonsmooth nonconvex optimization and demonstrates its practical advantages through extensive experiments.

Findings

Permutation randomization disrupts optimizer shrinkage behavior.

It enables convergence to the global optimum with enough iterations.

Permutation randomization preserves the convergence rate of the original optimizer.

Abstract

While gradient-based optimizers that incorporate randomization often showcase superior performance on complex optimization, the theoretical foundations underlying this superiority remain insufficiently understood. A particularly pressing question has emerged: What is the role of randomization in dimension-free nonsmooth nonconvex optimization? To address this gap, we investigate the theoretical and empirical impact of permutation randomization within gradient-based optimization frameworks, using it as a representative case to explore broader implications. From a theoretical perspective, our analyses reveal that permutation randomization disrupts the shrinkage behavior of gradient-based optimizers, facilitating continuous convergence toward the global optimum given a sufficiently large number of iterations. Additionally, we prove that permutation randomization can preserve the convergence rate of the underlying optimizer. On the empirical side, we conduct extensive numerical experiments comparing permutation-randomized optimizer against three baseline methods. These experiments span tasks such as training deep neural networks with stacked architectures and optimizing noisy objective functions. The results not only corroborate our theoretical insights but also highlight the practical benefits of permutation randomization. In summary, this work delivers both rigorous theoretical justification and compelling empirical evidence for the effectiveness of permutation randomization. Our findings and evidence lay a foundation for extending analytics to encompass a wide array of randomization.