The Influence of Nuisance Parameter Uncertainty on Statistical Inference in Practical Data Science Models

TL;DR

This paper explores how uncertainty in nuisance parameters affects statistical inference in data science models, proposing a method to adjust confidence regions for more accurate uncertainty quantification.

Contribution

It introduces a general procedure to modify confidence regions considering nuisance parameter uncertainty, validated on GARCH models and neural networks.

Findings

Adjusted confidence regions better reflect true uncertainty.

Method applicable to time series and neural network models.

Enhances reliability of statistical inference in practical models.

Abstract

For multiple reasons -- such as avoiding overtraining from one data set or because of having received numerical estimates for some parameters in a model from an alternative source -- it is sometimes useful to divide a model's parameters into one group of primary parameters and one group of nuisance parameters. However, uncertainty in the values of nuisance parameters is an inevitable factor that impacts the model's reliability. This paper examines the issue of uncertainty calculation for primary parameters of interest in the presence of nuisance parameters. We illustrate a general procedure on two distinct model forms: 1) the GARCH time series model with univariate nuisance parameter and 2) multiple hidden layer feed-forward neural network models with multivariate nuisance parameters. Leveraging an existing theoretical framework for nuisance parameter uncertainty, we show how to modify the confidence regions for the primary parameters while considering the inherent uncertainty introduced by nuisance parameters. Furthermore, our study validates the practical effectiveness of adjusted confidence regions that properly account for uncertainty in nuisance parameters. Such an adjustment helps data scientists produce results that more honestly reflect the overall uncertainty.