Natural Gradient for Combined Loss Using Wavelets

TL;DR

This paper introduces a novel natural gradient algorithm that employs wavelets to efficiently optimize a convex combination of different loss functionals, demonstrating improved performance through numerical experiments.

Contribution

It presents a new natural gradient method using wavelets to approximate the Hessian for combined loss functionals, enhancing optimization efficiency.

Findings

The wavelet-based approach effectively diagonalizes the Hessian.

Numerical results show improved convergence speed.

The method is applicable to various convex loss combinations.

Abstract

Natural gradients have been widely used in optimization of loss functionals over probability space, with important examples such as Fisher-Rao gradient descent for Kullback-Leibler divergence, Wasserstein gradient descent for transport-related functionals, and Mahalanobis gradient descent for quadratic loss functionals. This note considers the situation in which the loss is a convex linear combination of these examples. We propose a new natural gradient algorithm by utilizing compactly supported wavelets to diagonalize approximately the Hessian of the combined loss. Numerical results are included to demonstrate the efficiency of the proposed algorithm.