# Practical Valid Inferences for the Two-Sample Binomial Problem

## TL;DR

This paper reviews and evaluates various exact non-asymptotic methods for testing differences between two binomial proportions, focusing on their validity, interpretability, and practical properties, highlighting the lack of a perfect method.

## Contribution

It provides a comprehensive comparison of existing exact inference methods for the two-sample binomial problem and offers recommendations based on prioritized properties.

## Key findings

- No single method satisfies all desirable properties.
- Compatibility between p-values and confidence intervals varies across methods.
- Recommendations depend on which properties are most important for the application.

## Abstract

Our interest is whether two binomial parameters differ, which parameter is larger, and by how much. This apparently simple problem was addressed by Fisher in the 1930's, and has been the subject of many review papers since then. Yet there continues to be new work on this issue and no consensus solution. Previous reviews have focused primarily on testing and the properties of validity and power, or primarily on confidence intervals, their coverage, and expected length. Here we evaluate both. For example, we consider whether a p-value and its matching confidence interval are compatible, meaning that the p-value rejects at level $\alpha$ if and only if the $1-\alpha$ confidence interval excludes all null parameter values. For focus, we only examine non-asymptotic inferences, so that most of the p-values and confidence intervals are valid (i.e., exact) by construction. Within this focus, we review different methods emphasizing many of the properties and interpretational aspects we desire from applied frequentist inference: validity, accuracy, good power, equivariance, compatibility, coherence, and parameterization and direction of effect. We show that no one method can meet all the desirable properties and give recommendations based on which properties are given more importance.