Accelerated Over-Relaxation Heavy-Ball Method: Achieving Global Accelerated Convergence with Broad Generalization

TL;DR

This paper introduces the AOR-HB method, a novel optimization algorithm that achieves global accelerated convergence for smooth strongly convex problems, extending the heavy-ball method with broad applicability and optimal complexity.

Contribution

The paper presents the first globally accelerated heavy-ball variant with provable convergence, extending to composite and min-max problems, and achieving optimal complexity bounds.

Findings

Achieves global accelerated convergence for smooth strongly convex problems.

Extends to composite convex and min-max optimization problems.

Provides optimal complexity bounds for the proposed method.

Abstract

The heavy-ball momentum method accelerates gradient descent with a momentum term but lacks accelerated convergence for general smooth strongly convex problems. This work introduces the Accelerated Over-Relaxation Heavy-Ball (AOR-HB) method, the first variant with provable global and accelerated convergence for such problems. AOR-HB closes a long-standing theoretical gap, extends to composite convex optimization and min-max problems, and achieves optimal complexity bounds. It offers three key advantages: (1) broad generalization ability, (2) potential to reshape acceleration techniques, and (3) conceptual clarity and elegance compared to existing methods.