Monotonic Properties of Completed Aggregates in Recursive Queries

TL;DR

This paper explores the monotonic properties of aggregates in recursive queries, demonstrating how certain classes of aggregates can be safely used in recursion to improve efficiency and scalability in BigData systems.

Contribution

It provides a formal semantics for aggregates in recursion, enabling their safe use in scalable BigData query systems, extending previous work on monotonicity and stratification.

Findings

Aggregates often exhibit monotonicity in set-containment lattice.

PreM property allows equivalence to stratified programs with post-constraints.

Formal semantics supports efficient implementation in existing Datalog and SQL systems.

Abstract

The use of aggregates in recursion enables efficient and scalable support for a wide range of BigData algorithms, including those used in graph applications, KDD applications, and ML applications, which have proven difficult to be expressed and supported efficiently in BigData systems supporting Datalog or SQL. The problem with these languages and systems is that, to avoid the semantic and computational issues created by non-monotonic constructs in recursion, they only allow programs that are stratified with respect to negation and aggregates. Now, while this crippling restriction is well-justified for negation, it is frequently unjustified for aggregates, since (i) aggregates are often monotonic in the standard lattice of set-containment, (ii) the PreM property guarantees that programs with extrema in recursion are equivalent to stratified programs where extrema are used as post-constraints, and (iii) any program computing any aggregates on sets of facts of predictable cardinality tantamounts to stratified programs where the precomputation of the cardinality of the set is followed by a stratum where recursive rules only use monotonic constructs. With (i) and (ii) covered in previous papers, this paper focuses on (iii) using examples of great practical interest. For such examples, we provide a formal semantics that is conducive to efficient and scalable implementations via well-known techniques such as semi-naive fixpoint currently supported by most Datalog and SQL3 systems.