Conservative Hypothesis Tests and Confidence Intervals using Importance Sampling

TL;DR

This paper introduces a simple correction method for importance sampling p-values that ensures valid hypothesis tests and confidence intervals, improving reliability in multiple testing and complex inference scenarios.

Contribution

It proposes a correction to importance sampling p-values that guarantees valid hypothesis testing and confidence interval construction using only a single Monte Carlo sample.

Findings

Corrected p-values are valid under the null hypothesis.

The method maintains nominal significance levels in Monte Carlo confidence intervals.

Applications include neurophysiological data analysis and logistic regression inference.

Abstract

Importance sampling is a common technique for Monte Carlo approximation, including Monte Carlo approximation of p-values. Here it is shown that a simple correction of the usual importance sampling p-values creates valid p-values, meaning that a hypothesis test created by rejecting the null when the p-value is <= alpha will also have a type I error rate <= alpha. This correction uses the importance weight of the original observation, which gives valuable diagnostic information under the null hypothesis. Using the corrected p-values can be crucial for multiple testing and also in problems where evaluating the accuracy of importance sampling approximations is difficult. Inverting the corrected p-values provides a useful way to create Monte Carlo confidence intervals that maintain the nominal significance level and use only a single Monte Carlo sample. Several applications are described, including accelerated multiple testing for a large neurophysiological dataset and exact conditional inference for a logistic regression model with nuisance parameters.