Local Compositional Complexity: How to Detect a Human-readable Messsage

TL;DR

This paper introduces a new framework for measuring data complexity based on structured and unstructured parts, with a focus on human communication signals, and demonstrates its effectiveness across multiple domains.

Contribution

It proposes the concept of local compositionality as a computable measure of message-like complexity and applies it to distinguish meaningful signals from noise.

Findings

Successfully differentiates signals with meaningful content from noise in auditory, visual, and text data.

Provides a framework that could aid in detecting messages in extraterrestrial signals.

Offers a method to quantify data complexity in a way that aligns with human communication structures.

Abstract

Data complexity is an important concept in the natural sciences and related areas, but lacks a rigorous and computable definition. In this paper, we focus on a particular sense of complexity that is high if the data is structured in a way that could serve to communicate a message. In this sense, human speech, written language, drawings, diagrams and photographs are high complexity, whereas data that is close to uniform throughout or populated by random values is low complexity. We describe a general framework for measuring data complexity based on dividing the shortest description of the data into a structured and an unstructured portion, and taking the size of the former as the complexity score. We outline an application of this framework in statistical mechanics that may allow a more objective characterisation of the macrostate and entropy of a physical system. Then, we derive a more precise and computable definition geared towards human communication, by proposing local compositionality as an appropriate specific structure. We demonstrate experimentally that this method can distinguish meaningful signals from noise or repetitive signals in auditory, visual and text domains, and could potentially help determine whether an extra-terrestrial signal contained a message.