Entropy of Ukrainian

TL;DR

This paper estimates the entropy of the Ukrainian language using a Shannon-inspired experiment with 184 volunteers, providing an upper bound and comparing it to large language models.

Contribution

First entropy measurement for Ukrainian using a Shannon-based experiment, with methods and code published for reproducibility.

Findings

Upper bound of Ukrainian entropy approximately 1.201 bits per character

Methods and code for entropy estimation are documented and publicly available

Comparison of Ukrainian entropy with performance of current large language models

Abstract

In natural language processing, the entropy of a language is a measure of its unpredictability and complexity. The first study on this subject was conducted by Claude Shannon in 1951. By having participants predict the next character in a sentence, he was able to approximate the entropy of the English language. Several follow-up studies by other authors have since been conducted for English, and one for Hebrew. However, to date, Shannon's experiment has never been conducted for Ukrainian. In this paper, we perform this experiment for Ukrainian by recruiting 184 volunteers using social media channels. We rely on techniques used for English to approximate the entropy value of Ukrainian. The final result is an upper bound of $H_{upper}\approx1.201$ bits per character. We compare this to the performance of current Large Language Models. The methods and code used are also documented and published, along with a discussion of the main challenges encountered.