Universal Bayesian Measures and Universal Histogram Sequences

TL;DR

This paper introduces a universal Bayesian measure that generalizes data compression and model estimation to sources with discrete, continuous, or mixed variables, overcoming limitations of histogram-based methods.

Contribution

It extends the concept of universal Bayesian measures to more general sources, improving model approximation and inference capabilities beyond traditional histogram approaches.

Findings

Constructed a universal Bayesian measure applicable to mixed variable types.

Overcame limitations of histogram range dependency in source modeling.

Extended applications of the minimum description length principle.

Abstract

Consider universal data compression: the length $l(x^n)$ of sequence $x^n\in A^n$ with finite alphabet $A$ and length $n$ satisfies Kraft's inequality over $A^n$, and $-\frac{1}{n}\log \frac{P^n(x^n)}{Q^n(x^n)}$ almost surely converges to zero as $n$ grows for the $Q^n(x^n)=2^{-l(x^n)}$ and any stationary ergodic source $P$. In this paper, we say such a $Q$ is a universal Bayesian measure. We generalize the notion to the sources in which the random variables may be either discrete, continuous, or none of them. The basic idea is due to Boris Ryabko who utilized model weighting over histograms that approximate $P$, assuming that a density function of $P$ exists. However, the range of $P$ depends on the choice of the histogram sequence. The universal Bayesian measure constructed in this paper overcomes the drawbacks and has many applications to infer relation among random variables, and extends the application area of the minimum description length principle.