Contextuality from missing and versioned data

TL;DR

This paper explores how the concept of contextuality, originally from quantum mechanics, can manifest in modern distributed data systems with missing and versioned data, highlighting fundamental inconsistencies.

Contribution

It introduces the idea that contextuality can occur in distributed data scenarios, extending the concept beyond quantum physics to data management challenges.

Findings

Contextuality can arise in distributed databases with missing data.

Versioning and snapshot isolation contribute to data contextuality.

The paper connects quantum contextuality to data inconsistency phenomena.

Abstract

Traditionally categorical data analysis (e.g. generalized linear models) works with simple, flat datasets akin to a single table in a database with no notion of missing data or conflicting versions. In contrast, modern data analysis must deal with distributed databases with many partial local tables that need not always agree. The computational agents tabulating these tables are spatially separated, with binding speed-of-light constraints and data arriving too rapidly for these distributed views ever to be fully informed and globally consistent. Contextuality is a mathematical property which describes a kind of inconsistency arising in quantum mechanics (e.g. in Bell's theorem). In this paper we show how contextuality can arise in common data collection scenarios, including missing data and versioning (as in low-latency distributed databases employing snapshot isolation). In the companion paper, we develop statistical models adapted to this regime.