Practical Perspectives on Symplectic Accelerated Optimization

TL;DR

This paper explores practical enhancements for symplectic accelerated optimization algorithms, focusing on momentum restarting, temporal looping, and geometric integration techniques to improve efficiency, stability, and ease of tuning.

Contribution

It introduces practical strategies like momentum restarting and temporal looping that significantly improve the robustness and simplicity of symplectic accelerated optimization algorithms.

Findings

Momentum restarting reduces oscillations and improves robustness.

Temporal looping prevents numerical instability without sacrificing efficiency.

Comparison of geometric integration methods informs better tuning practices.

Abstract

Geometric numerical integration has recently been exploited to design symplectic accelerated optimization algorithms by simulating the Lagrangian and Hamiltonian systems from the variational framework introduced in Wibisono et al. In this paper, we discuss practical considerations which can significantly boost the computational performance of these optimization algorithms, and considerably simplify the tuning process. In particular, we investigate how momentum restarting schemes ameliorate computational efficiency and robustness by reducing the undesirable effect of oscillations, and ease the tuning process by making time-adaptivity superfluous. We also discuss how temporal looping helps avoiding instability issues caused by numerical precision, without harming the computational efficiency of the algorithms. Finally, we compare the efficiency and robustness of different geometric integration techniques, and study the effects of the different parameters in the algorithms to inform and simplify tuning in practice. From this paper emerge symplectic accelerated optimization algorithms whose computational efficiency, stability and robustness have been improved, and which are now much simpler to use and tune for practical applications.