Uncovering Bias in Order Assignment

TL;DR

This paper introduces three new statistical tests to assess whether the orderings in various real-world scenarios are truly random, revealing significant deviations in political ballots that could have legal implications.

Contribution

It develops untargeted, no-a priori-information tests for randomness in order sequences and applies them to diverse real-world data, uncovering non-random patterns.

Findings

Deviations from randomness detected in election ballots

Lottery number draws show no significant bias

Contestant performance order appears random

Abstract

Many real life situations require a set of items to be repeatedly placed in a random sequence. In such circumstances, it is often desirable to test whether such randomization indeed obtains, yet this problem has received very limited attention in the literature. This paper articulates the key features of this problem and presents three "untargeted" tests that require no a priori information from the analyst. These methods are used to analyze the order in which lottery numbers are drawn in Powerball, the order in which contestants perform on American Idol, and the order of candidates on primary election ballots in Texas and West Virginia. In this last application, multiple deviations from full randomization are detected, with potentially serious political and legal consequences. The form these deviations take varies, depending on institutional factors, which sometimes necessitates the use of tests that exchange power for increased robustness.