Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks

TL;DR

This paper evaluates the performance of various optimization algorithms on source code-related deep learning tasks, revealing significant differences in effectiveness and highlighting RAdam as a consistently strong choice.

Contribution

It provides the first comprehensive benchmark of optimizers on ML4SE tasks, demonstrating RAdam's superior performance and urging the community to reconsider default optimizer choices.

Findings

Optimizer choice significantly affects model quality, with up to two-fold score differences.

RAdam and its Lookahead variant outperform other optimizers on source code tasks.

The study highlights the need for further research on optimizer performance in code-related machine learning.

Abstract

Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a hundred different optimizers, yet there is scarce data on optimizer performance in various tasks. In particular, none of the benchmarks test the performance of optimizers on source code-related problems. However, existing benchmark data indicates that certain optimizers may be more efficient for particular domains. In this work, we test the performance of various optimizers on deep learning models for source code and find that the choice of an optimizer can have a significant impact on the model quality, with up to two-fold score differences between some of the relatively well-performing optimizers. We also find that RAdam optimizer (and its modification with the Lookahead envelope) is the best optimizer that almost always performs well on the tasks we consider. Our findings show a need for a more extensive study of the optimizers in code-related tasks, and indicate that the ML4SE community should consider using RAdam instead of Adam as the default optimizer for code-related deep learning tasks.