On the use of hybrid coarse-level models in multilevel minimization methods

TL;DR

This paper explores hybrid coarse-level models in multilevel minimization, combining additive and multiplicative approaches to improve convergence speed in large-scale nonlinear problems, especially in deep learning applications.

Contribution

It introduces a hybrid methodology for coarse-level models in NMM, enhancing convergence efficiency over traditional additive models.

Findings

Hybrid approach significantly speeds up convergence.

Numerical experiments demonstrate improved scalability.

Hybrid models outperform purely additive models in deep learning tasks.

Abstract

Solving large-scale nonlinear minimization problems is computationally demanding. Nonlinear multilevel minimization (NMM) methods explore the structure of the underlying minimization problem to solve such problems in a computationally efficient and scalable manner. The efficiency of the NMM methods relies on the quality of the coarse-level models. Traditionally, coarse-level models are constructed using the additive approach, where the so-called $\tau$-correction enforces a local coherence between the fine-level and coarse-level objective functions. In this work, we extend this methodology and discuss how to enforce local coherence between the objective functions using a multiplicative approach. Moreover, we also present a hybrid approach, which takes advantage of both, additive and multiplicative, approaches. Using numerical experiments from the field of deep learning, we show that employing a hybrid approach can greatly improve the convergence speed of NMM methods and therefore it provides an attractive alternative to the almost universally used additive approach.