Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimization

TL;DR

Alada is an adaptive momentum optimization method that efficiently handles large-scale matrix and tensor variables by using a rank-one factorization approach, reducing memory usage while maintaining robustness and performance.

Contribution

This paper introduces Alada, a novel adaptive momentum method employing rank-one factorization for memory-efficient stochastic optimization of matrices and tensors.

Findings

Reduces memory overhead compared to Adam.

Maintains comparable theoretical performance to traditional methods.

Demonstrates robustness and efficiency in NLP tasks.

Abstract

This work proposes Alada, an adaptive momentum method for stochastic optimization over large-scale matrices. Alada employs a rank-one factorization approach to estimate the second moment of gradients, where factors are updated alternatively to minimize the estimation error. Alada achieves sublinear memory overheads and can be readily extended to optimizing tensor-shaped variables.We also equip Alada with a first moment estimation rule, which enhances the algorithm's robustness without incurring additional memory overheads. The theoretical performance of Alada aligns with that of traditional methods such as Adam. Numerical studies conducted on several natural language processing tasks demonstrate the reduction in memory overheads and the robustness in training large models relative to Adam and its variants.