Bounds on Bayes Factors for Binomial A/B Testing

TL;DR

This paper establishes bounds on Bayes factors in binomial A/B testing, linking them to Jensen-Shannon divergence and Welch's statistic, thus bridging Bayesian and frequentist approaches with information geometric tools.

Contribution

It introduces bounds on Bayes factors in binomial A/B testing based on divergence measures, connecting Bayesian and frequentist methods innovatively.

Findings

Bayes factors are controlled by Jensen-Shannon divergence.

Bayesian sample bounds closely match frequentist bounds.

The approach uses information geometry tools for derivation.

Abstract

Bayes factors, in many cases, have been proven to bridge the classic -value based significance testing and bayesian analysis of posterior odds. This paper discusses this phenomena within the binomial A/B testing setup (applicable for example to conversion testing). It is shown that the bayes factor is controlled by the \emph{Jensen-Shannon divergence} of success ratios in two tested groups, which can be further bounded by the Welch statistic. As a result, bayesian sample bounds almost match frequentionist's sample bounds. The link between Jensen-Shannon divergence and Welch's test as well as the derivation are an elegant application of tools from information geometry.