Exact Probability Distribution versus Entropy

TL;DR

This paper investigates the average number of guesses needed to identify a word based on letter probabilities, providing approximations and analytical expressions that relate to entropy and distribution properties.

Contribution

It introduces new approximation methods for estimating guesses in large alphabets and word lengths, including analytical formulas for normal-like probability distributions.

Findings

Approximate guesses within minutes for realistic sizes

Logarithm density of probabilities is often normal

Guessing proportion decreases exponentially with word length

Abstract

The problem addressed concerns the determination of the average number of successive attempts of guessing a word of a certain length consisting of letters with given probabilities of occurrence. Both first- and second-order approximations to a natural language are considered. The guessing strategy used is guessing words in decreasing order of probability. When word and alphabet sizes are large, approximations are necessary in order to estimate the number of guesses. Several kinds of approximations are discussed demonstrating moderate requirements concerning both memory and CPU time. When considering realistic sizes of alphabets and words (100) the number of guesses can be estimated within minutes with reasonable accuracy (a few percent). For many probability distributions the density of the logarithm of probability products is close to a normal distribution. For those cases it is possible to derive an analytical expression for the average number of guesses. The proportion of guesses needed on average compared to the total number decreases almost exponentially with the word length. The leading term in an asymptotic expansion can be used to estimate the number of guesses for large word lengths. Comparisons with analytical lower bounds and entropy expressions are also provided.