ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization

TL;DR

This paper introduces ZO-AdaMM, a zeroth-order adaptive momentum method for black-box optimization, demonstrating faster convergence than existing methods and applications in adversarial attacks.

Contribution

We propose ZO-AdaMM, extending AdaMM to gradient-free settings, analyze its convergence, and apply it to black-box neural network attacks.

Findings

ZO-AdaMM converges faster than 6 state-of-the-art ZO methods on ImageNet.

Convergence rate is roughly O(√d) worse than first-order AdaMM.

Mahalanobis distance is crucial for convergence analysis.

Abstract

The adaptive momentum method (AdaMM), which uses past gradients to update descent directions and learning rates simultaneously, has become one of the most popular first-order optimization methods for solving machine learning problems. However, AdaMM is not suited for solving black-box optimization problems, where explicit gradient forms are difficult or infeasible to obtain. In this paper, we propose a zeroth-order AdaMM (ZO-AdaMM) algorithm, that generalizes AdaMM to the gradient-free regime. We show that the convergence rate of ZO-AdaMM for both convex and nonconvex optimization is roughly a factor of $O(\sqrt{d})$ worse than that of the first-order AdaMM algorithm, where $d$ is problem size. In particular, we provide a deep understanding on why Mahalanobis distance matters in convergence of ZO-AdaMM and other AdaMM-type methods. As a byproduct, our analysis makes the first step toward understanding adaptive learning rate methods for nonconvex constrained optimization. Furthermore, we demonstrate two applications, designing per-image and universal adversarial attacks from black-box neural networks, respectively. We perform extensive experiments on ImageNet and empirically show that ZO-AdaMM converges much faster to a solution of high accuracy compared with $6$ state-of-the-art ZO optimization methods.