Structured second-order methods via natural gradient descent

TL;DR

This paper introduces structured second-order and adaptive-gradient methods based on natural-gradient descent, offering invariant properties and simplicity, with demonstrated efficiency on non-convex and deep learning tasks.

Contribution

It develops new structured second-order and adaptive-gradient algorithms using natural-gradient descent with invariant properties and simple expressions.

Findings

Effective on deterministic non-convex problems

Demonstrated efficiency in deep learning tasks

Structured methods maintain invariance and simplicity

Abstract

In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces. Natural-gradient descent is an attractive approach to design new algorithms in many settings such as gradient-free, adaptive-gradient, and second-order methods. Our structured methods not only enjoy a structural invariance but also admit a simple expression. Finally, we test the efficiency of our proposed methods on both deterministic non-convex problems and deep learning problems.