Enhanced Monte Carlo Estimation of the Fisher Information Matrix with Independent Perturbations for Complex Problems

TL;DR

This paper introduces an improved Monte Carlo method with independent perturbations for more accurate Fisher Information Matrix estimation in complex models, reducing variance and balancing computational costs.

Contribution

It presents a novel resampling-based approach with independent perturbations, enhancing variance reduction in FIM estimation compared to existing methods.

Findings

Variance reduction from O(1/N) to O(1/(nN))

Theoretical analysis confirms improved accuracy

Numerical experiments demonstrate effectiveness

Abstract

The Fisher information matrix provides a way to measure the amount of information given observed data based on parameters of interest. Many applications of the FIM exist in statistical modeling, system identification, and parameter estimation. We sometimes use the Monte Carlo-based method to estimate the FIM because its analytical form is often impossible or difficult to be computed in real-world models. In this paper, we review the basic method based on simultaneous perturbations and present an enhanced resampling-based method with independent simultaneous perturbations to estimate the Fisher information matrix. We conduct theoretical and numerical analysis to show its accuracy via variance reduction from $O(1/N)$ to $O(1/(nN))$, where $n$ is the sample size of the data and $N$ is a measure of the Monte Carlo averaging. We also consider the trade-off between accuracy and computational cost.