Experiments on predictability of word in context and information rate in natural language

TL;DR

This study investigates how the unpredictability of words in context relates to their length, suggesting natural language maintains a consistent information rate across different text forms.

Contribution

It provides empirical evidence that word unpredictability correlates linearly with word length, supporting the hypothesis of a constant information rate in natural language.

Findings

Unpredictability depends linearly on word length in both poetry and prose.

The effect persists even when word length is unknown to subjects.

Supports the idea of an even information rate in natural language.

Abstract

Based on data from a large-scale experiment with human subjects, we conclude that the logarithm of probability to guess a word in context (unpredictability) depends linearly on the word length. This result holds both for poetry and prose, even though with prose, the subjects don't know the length of the omitted word. We hypothesize that this effect reflects a tendency of natural language to have an even information rate.