Estimating accurate covariance matrices on fitted model parameters

TL;DR

This paper presents algorithms for accurately estimating covariance matrices of fitted model parameters using numerical differentiation of the Hessian, with validation on datasets where analytical solutions are available.

Contribution

It introduces methods for computing precise covariance matrices from the Hessian, including for posterior distributions, improving upon neglected aspects in statistical modeling.

Findings

Numerical differentiation yields highly accurate covariance matrices.

Algorithms perform well on datasets with known analytical Hessians.

Enhanced covariance estimation improves statistical inference accuracy.

Abstract

The accurate computation of the covariance matrix of fitted model parameters is a somewhat neglected task in Statistics. Algorithms are given for computing accurate covariance matrices derived from computing the Hessian matrix by numerical differentiation, and also for the covariance matrix of the posterior distribution of model parameters. Evaluations on two datasets where the Hessian could be computed analytically show that the numerical differentiation algorithm is very accurate.