Two ways game-theoretic probability can improve data analysis

TL;DR

This paper explores how game-theoretic probability offers a flexible framework for hypothesis testing and descriptive data analysis, allowing optional data collection and honest comparison of distributions.

Contribution

It introduces a game-theoretic approach to hypothesis testing and data analysis that extends traditional methods with optional continuation and betting strategies.

Findings

Game-theoretic probability supports optional data collection during testing.

Betting-based methods can replace traditional confidence intervals with likelihood sets.

The approach simplifies and generalizes hypothesis testing and descriptive analysis.

Abstract

When testing a statistical hypothesis, is it legitimate to deliberate on the basis of initial data about whether and how to collect further data? Game-theoretic probability's fundamental principle for testing by betting says yes, provided that you are testing by betting and do not risk more capital than initially committed. Standard statistical theory uses Cournot's principle, which does not allow such optional continuation. Cournot's principle can be extended to allow optional continuation when testing is carried out by multiplying likelihood ratios, but the extension lacks the simplicity and generality of testing by betting. Game-theoretic probability can also help us with descriptive data analysis. To obtain a purely and honestly descriptive analysis using competing probability distributions, we have them bet against each other using the Kelly principle. The place of confidence intervals is then taken by a sets of distributions that do relatively well in the competition. In the simplest implementation, these sets coincide with R. A. Fisher's likelihood intervals.