Querying Incomplete Numerical Data: Between Certain and Possible Answers

TL;DR

This paper develops a probabilistic framework for handling incomplete numerical data in databases, enabling meaningful query answers with arithmetic and aggregation by modeling missing values with probability distributions.

Contribution

It introduces a principled probabilistic model for numerical nulls, providing a compositional query answering framework with efficient approximation algorithms.

Findings

Queries are measurable and have finite representations.

Probability intervals offer meaningful answers where classical methods fail.

Approximation algorithms efficiently estimate probabilities for numerical query answers.

Abstract

Queries with aggregation and arithmetic operations, as well as incomplete data, are common in real-world database, but we lack a good understanding of how they should interact. On the one hand, systems based on SQL provide ad-hoc rules for numerical nulls, on the other, theoretical research largely concentrates on the standard notions of certain and possible answers. In the presence of numerical attributes and aggregates, however, these answers are often meaningless, returning either too little or too much. Our goal is to define a principled framework for databases with numerical nulls and answering queries with arithmetic and aggregations over them. Towards this goal, we assume that missing values in numerical attributes are given by probability distributions associated with marked nulls. This yields a model of probabilistic bag databases in which tuples are not necessarily independent, since nulls can repeat. We provide a general compositional framework for query answering, and then concentrate on queries that resemble standard SQL with arithmetic and aggregation. We show that these queries are measurable, and that their outputs have a finite representation. Moreover, since the classical forms of answers provide little information in the numerical setting, we look at the probability that numerical values in output tuples belong to specific intervals. Even though their exact computation is intractable, we show efficient approximation algorithms to compute such probabilities.