Watching a drunkard for ten nights: A study of distributions of variances

TL;DR

This paper investigates the full probability distribution of variances in random walk data to understand surprising results and their implications in statistical analysis.

Contribution

It provides a detailed analysis of the variance distribution in random walks, highlighting the importance of full distributions over just mean and standard deviation.

Findings

Full variance distribution can yield complex, nonsensical estimates of probabilities.

Analyzing the full distribution helps interpret surprising statistical results.

The study connects variance distribution to data binning and other examples.

Abstract

For any physical observable in statistical systems, the most frequently studied quantities are its average and standard deviation. Yet, its full distribution often carries extremely interesting information and can be invoked to put any surprising properties of the individual moments into perspective. As an example, we consider a problem concerning simple random walks which was posed in a recent text. When a drunk is observed over L nights, taking N steps per night, and the number of steps to the right is recorded for each night, an average and a variance based on these data can be computed. When the variance is used to estimate p, the probability for the drunk to step right, complex values for p are frequently found. To put such obviously nonsensical results into context, we study the full probability distribution for the variance of the data string. We discuss the connection of our results to the problem of data binning and provide two other brief examples to demonstrate the importance of full distributions.