Breakdown of statistical inference from some random experiments

TL;DR

This paper reveals that standard statistical inference can be misleading when applied to large, inhomogeneous samples from pseudo-random experiments, emphasizing the importance of homogeneity tests like chi-square.

Contribution

It demonstrates the failure of traditional asymptotic methods on inhomogeneous data and advocates for incorporating homogeneity tests to ensure valid statistical conclusions.

Findings

Large samples can be highly misleading if inhomogeneous.

Chi-square tests effectively detect sample inhomogeneity.

Standard methods underestimate statistical errors without homogeneity checks.

Abstract

Many experiments can be interpreted in terms of random processes operating according to some internal protocols. When experiments are costly or cannot be repeated only one or a few finite samples are available. In this paper we study data generated by pseudo-random computer experiments operating according to particular internal protocols. We show that the standard statistical analysis performed on a sample, containing 100000 data points or more, may sometimes be highly misleading and statistical errors largely underestimated. Our results confirm in a dramatic way the dangers of standard asymptotic statistical inference if a sample is not homogenous. We demonstrate that analyzing various subdivisions of samples by multiple chi-square tests and chi-square frequency graphs is very effective in detecting sample inhomogeneity. Therefore to assure correctness of the statistical inference the above mentioned chi-square tests and other non-parametric sample homogeneity tests should be incorporated in any statistical analysis of experimental data. If such tests are not performed the reported conclusions and estimates of the errors cannot be trusted.