Asymptotically Minimax Regret by Bayes Mixtures

TL;DR

This paper investigates the minimax regret in data compression, gambling, and prediction, demonstrating that Bayesian mixtures with Jeffreys priors achieve optimal asymptotic performance across various exponential and non-exponential families.

Contribution

It introduces a modified Jeffreys prior for non-exponential families and characterizes minimax regret for complex models like mixture and curved families.

Findings

Bayesian mixtures with Jeffreys priors achieve asymptotic minimax regret.

Modified Jeffreys prior extends to non-exponential families using local exponential tilting.

Results apply to mixture, curved, and contamination models, characterizing Rissanen's stochastic complexity.

Abstract

We study the problems of data compression, gambling and prediction of a sequence $x^n=x_1x_2...x_n$ from an alphabet ${\cal X}$, in terms of regret and expected regret (redundancy) with respect to various smooth families of probability distributions. We evaluate the regret of Bayes mixture distributions compared to maximum likelihood, under the condition that the maximum likelihood estimate is in the interior of the parameter space. For general exponential families (including the non-i.i.d.\ case) the asymptotically mimimax value is achieved when variants of the prior of Jeffreys are used. %under the condition that the maximum likelihood estimate is in the interior of the parameter space. Interestingly, we also obtain a modification of Jeffreys prior which has measure outside the given family of densities, to achieve minimax regret with respect to non-exponential type families. This modification enlarges the family using local exponential tilting (a fiber bundle). Our conditions are confirmed for certain non-exponential families, including curved families and mixture families (where either the mixture components or their weights of combination are parameterized) as well as contamination models. Furthermore for mixture families we show how to deal with the full simplex of parameters. These results also provide characterization of Rissanen's stochastic complexity.