An introduction to how chi-square and classical exact tests often wildly misreport significance and how the remedy lies in computers

TL;DR

This paper highlights the limitations of traditional chi-square and exact tests in goodness-of-fit testing for discrete distributions and advocates for Euclidean-based tests, which are more powerful and now practically computable with modern software.

Contribution

It demonstrates that Euclidean distance-based goodness-of-fit tests outperform classical chi-square and exact tests, especially for non-uniform distributions, and emphasizes the role of computers in implementing these tests.

Findings

Euclidean-based tests outperform classical tests by at least an order of magnitude.

Modern computers enable practical calculation of Euclidean goodness-of-fit significance.

Euclidean tests are more reliable for non-uniform discrete distributions.

Abstract

Goodness-of-fit tests based on the Euclidean distance often outperform chi-square and other classical tests (including the standard exact tests) by at least an order of magnitude when the model being tested for goodness-of-fit is a discrete probability distribution that is not close to uniform. The present article discusses numerous examples of this. Goodness-of-fit tests based on the Euclidean metric are now practical and convenient: although the actual values taken by the Euclidean distance and similar goodness-of-fit statistics are seldom humanly interpretable, black-box computer programs can rapidly calculate their precise significance.