On the incorrect use of Carlisle's method for dichotomous variables

TL;DR

This paper critiques the misuse of Carlisle's method for dichotomous variables in fraud detection, demonstrating its inaccuracies and proposing a correction to improve its validity.

Contribution

It clarifies the improper adaptation of Carlisle's method for dichotomous data and offers a simple correction to ensure accurate p-value calculations.

Findings

Naive adaptation leads to significantly incorrect p-values

Proposed correction improves the method's accuracy

Clarifies proper application of Carlisle's method for different data types

Abstract

In 2017, J. B. Carlisle has proposed a method for fraud detection in randomized controlled trials based on a comparison of reported baseline data between treatment groups. While Carlisle has only used the method for continuous variables, some authors have recently employed a naive adaption of the method for dichotomous variables. We explain why such adaptation leads to p-values that are wrong by orders of magnitude and we make a simple concrete proposal for correction of the method.