The Jacobian and Hessian of the Kullback-Leibler Divergence between Multivariate Gaussian Distributions (Technical Report)

TL;DR

This technical report provides detailed derivations of the Jacobian and Hessian matrices of the Kullback-Leibler divergence between two multivariate Gaussian distributions, enhancing understanding of their differential properties.

Contribution

It offers a comprehensive, didactic derivation of the Jacobian and Hessian matrices for the KL divergence between Gaussian distributions, based on established differential theory.

Findings

Explicit formulas for Jacobian and Hessian matrices

Step-by-step derivations with detailed explanations

Enhanced understanding of differential properties of KL divergence

Abstract

This document shows how to obtain the Jacobian and Hessian matrices of the Kullback-Leibler divergence between two multivariate Gaussian distributions, using the first and second-order differentials. The presented derivations are based on the theory presented by \cite{magnus99}. I've also got great inspiration from some of the derivations in \cite{minka}. Since I pretend to be at most didactic, the document is split into a summary of results and detailed derivations on each of the elements involved, with specific references to the tricks used in the derivations, and to many of the underlying concepts.