Dual Riemannian Newton Method on Statistical Manifolds

TL;DR

This paper introduces a second-order optimization method on statistical manifolds that leverages dual affine connections, achieving faster convergence by respecting the underlying information geometry.

Contribution

It proposes the dual Riemannian Newton method, a novel second-order algorithm that incorporates dual connections for improved optimization on statistical manifolds.

Findings

Establishes local quadratic convergence of the method.

Demonstrates effectiveness through experiments on statistical models.

Highlights the importance of dual connections in second-order optimization.

Abstract

In probabilistic modeling, parameter estimation is commonly formulated as a minimization problem on a parameter manifold. Optimization in such spaces requires geometry-aware methods that respect the underlying information structure. While the natural gradient leverages the Fisher information metric as a form of Riemannian gradient descent, it remains a first-order method and often exhibits slow convergence near optimal solutions. Existing second-order manifold algorithms typically rely on the Levi-Civita connection, thus overlooking the dual-connection structure that is central to information geometry. We propose the dual Riemannian Newton method, a Newton-type optimization algorithm on manifolds endowed with a metric and a pair of dual affine connections. The dual Riemannian Newton method explicates how duality shapes second-order updates: when the retraction (a local surrogate of the exponential map) is defined by one connection, the associated Newton equation is posed with its dual. We establish local quadratic convergence and validate the theory with experiments on representative statistical models. Thus, the dual Riemannian Newton method thus delivers second-order efficiency while remaining compatible with the dual structures that underlie modern information-geometric learning and inference.