Reallocating and Resampling: A Comparison for Inference

TL;DR

This paper compares reallocating and resampling methods in simulation-based inference, examining their effectiveness for testing and estimation across different data collection scenarios, revealing when each method is preferable.

Contribution

It provides a systematic comparison of reallocating and resampling, clarifying their relative advantages for inference under various data collection methods.

Findings

Reallocating outperforms resampling in small-sample hypothesis testing.

Resampling generally yields better estimates unless effects are additive.

Both methods are asymptotically equivalent for testing.

Abstract

Simulation-based inference plays a major role in modern statistics, and often employs either reallocating (as in a randomization test) or resampling (as in bootstrapping). Reallocating mimics random allocation to treatment groups, while resampling mimics random sampling from a larger population; does it matter whether the simulation method matches the data collection method? Moreover, do the results differ for testing versus estimation? Here we answer these questions in a simple setting by exploring the distribution of a sample difference in means under a basic two group design and four different scenarios: true random allocation, true random sampling, reallocating, and resampling. For testing a sharp null hypothesis, reallocating is superior in small samples, but reallocating and resampling are asymptotically equivalent. For estimation, resampling is generally superior, unless the effect is truly additive. Moreover, these results hold regardless of whether the data were collected by random sampling or random allocation.