Optimal algorithms for universal random number generation from finite memory sources

TL;DR

This paper develops optimal universal algorithms for random number generation from finite memory sources, generalizing Elias's scheme and introducing new methods for fixed-to-variable and variable-to-fixed regimes, including the twice-universal case.

Contribution

It characterizes the essentially unique optimal universal RNGs for finite memory sources, extending existing schemes and analyzing their expected lengths with second-order precision.

Findings

Optimal universal RNGs maximize expected output length in FVR and minimize input length in VFR.

The schemes are generalizations of Elias's method and include a novel VFR scheme.

Expected lengths are characterized up to an additive constant, including second-order terms.

Abstract

We study random number generators (RNGs), both in the fixed to variable-length (FVR) and the variable to fixed-length (VFR) regimes, in a universal setting in which the input is a finite memory source of arbitrary order and unknown parameters, with arbitrary input and output (finite) alphabet sizes. Applying the method of types, we characterize essentially unique optimal universal RNGs that maximize the expected output (respectively, minimize the expected input) length in the FVR (respectively, VFR) case. For the FVR case, the RNG studied is a generalization of Elias's scheme, while in the VFR case the general scheme is new. We precisely characterize, up to an additive constant, the corresponding expected lengths, which include second-order terms similar to those encountered in universal data compression and universal simulation. Furthermore, in the FVR case, we consider also a "twice-universal" setting, in which the Markov order k of the input source is also unknown.