# A Formalization of The Natural Gradient Method for General Similarity Measures

## TL;DR

This paper extends the natural gradient method to arbitrary similarity measures between distributions by formalizing a general framework for deriving appropriate metrics, enhancing optimization efficiency beyond likelihood maximization.

## Contribution

It introduces a novel framework for defining natural gradients for any similarity measure, connecting it with existing methods and enabling new applications.

## Key findings

- Derived natural gradients for various similarity measures
- Established connections with existing optimization techniques
- Demonstrated the framework with computational examples

## Abstract

In optimization, the natural gradient method is well-known for likelihood maximization. The method uses the Kullback-Leibler divergence, corresponding infinitesimally to the Fisher-Rao metric, which is pulled back to the parameter space of a family of probability distributions. This way, gradients with respect to the parameters respect the Fisher-Rao geometry of the space of distributions, which might differ vastly from the standard Euclidean geometry of the parameter space, often leading to faster convergence. However, when minimizing an arbitrary similarity measure between distributions, it is generally unclear which metric to use. We provide a general framework that, given a similarity measure, derives a metric for the natural gradient. We then discuss connections between the natural gradient method and multiple other optimization techniques in the literature. Finally, we provide computations of the formal natural gradient to show overlap with well-known cases and to compute natural gradients in novel frameworks.