Zero-disparity Distribution Synthesis: Fast Exact Calculation of Chi-Squared Statistic Distribution for Discrete Uniform Histograms

TL;DR

This paper introduces Zero-disparity Distribution Synthesis, a dynamic programming method for efficiently computing the exact chi-squared distribution for discrete histograms, improving accuracy over traditional approximations especially with low counts.

Contribution

It presents a novel fast algorithm for exact chi-squared distribution calculation, revealing limitations of approximation methods and enabling more precise statistical analysis.

Findings

Exact distribution computation exposes approximation errors.

The method reveals subtle pitfalls in traditional chi-squared tests.

Provides a Python implementation for practical use.

Abstract

Pearson's chi-squared test is widely used to assess the uniformity of discrete histograms, typically relying on a continuous chi-squared distribution to approximate the test statistic, since computing the exact distribution is computationally too costly. While effective in many cases, this approximation allegedly fails when expected bin counts are low or tail probabilities are needed. Here, Zero-disparity Distribution Synthesis is presented, a fast dynamic programming approach for computing the exact distribution, enabling detailed analysis of approximation errors. The results dispel some existing misunderstandings and also reveal subtle, but significant pitfalls in approximation that are only apparent with exact values. The Python source code is available at https://github.com/DiscreteTotalVariation/ChiSquared.