Severe testing of Benford's law

TL;DR

This paper examines the limitations of traditional statistical tests for Benford's law, proposing severity testing and deriving new asymptotic distributions to better assess data conformity, especially in large samples.

Contribution

It introduces severity testing for Benford's law, derives the asymptotic distribution of the MAD statistic, and applies these methods to controversial datasets.

Findings

Traditional tests may reject Benford's law in large samples despite minor deviations

Asymptotic distribution of MAD statistic is derived for better assessment

Severity testing provides a more nuanced evaluation of data conformity

Abstract

Benford's law is often used as a support to critical decisions related to data quality or the presence of data manipulations or even fraud. However, many authors argue that conventional statistical tests will reject the null of data "Benford-ness" if applied in samples of the typical size in this kind of applications, even in the presence of tiny and practically unimportant deviations from Benford's law. Therefore, they suggest using alternative criteria that, however, lack solid statistical foundations. This paper contributes to the debate on the "large $n$" (or "excess power") problem in the context of Benford's law testing. This issue is discussed in relation with the notion of severity testing for goodness of fit tests, with a specific focus on tests for conformity with Benford's law. To do so, we also derive the asymptotic distribution of the mean absolute deviation ($MAD$) statistic as well as an asymptotic standard normal test. Finally, the severity testing principle is applied to six controversial data sets to assess their "Benford-ness".