Word statistics in Blogs and RSS feeds: Towards empirical universal evidence

TL;DR

This paper analyzes word occurrence statistics in Blogs and RSS feeds, revealing universal patterns and deviations from Poisson processes, with implications for understanding language dynamics online.

Contribution

It introduces a method to study word statistics by grouping words by frequency and demonstrates universal features in waiting time distributions across different word classes.

Findings

Waiting times follow a stretched exponential distribution.

Deviations from Poisson statistics indicate non-trivial correlations.

Universal features are observed across different word frequency classes.

Abstract

We focus on the statistics of word occurrences and of the waiting times between such occurrences in Blogs. Due to the heterogeneity of words' frequencies, the empirical analysis is performed by studying classes of "frequently-equivalent" words, i.e. by grouping words depending on their frequencies. Two limiting cases are considered: the dilute limit, i.e. for those words that are used less than once a day, and the dense limit for frequent words. In both cases, extreme events occur more frequently than expected from the Poisson hypothesis. These deviations from Poisson statistics reveal non-trivial time correlations between events that are associated with bursts of activities. The distribution of waiting times is shown to behave like a stretched exponential and to have the same shape for different sets of words sharing a common frequency, thereby revealing universal features.