Optimizing Differentially-Maintained Recursive Queries on Dynamic Graphs

TL;DR

This paper enhances the scalability of differential computation for recursive queries on dynamic graphs by proposing optimizations that reduce memory overhead through dropping and recomputing differences, supported by experimental validation.

Contribution

It introduces novel optimizations for differential computation that significantly decrease memory usage in recursive query processing on dynamic graphs.

Findings

Memory overhead is reduced by dropping differences.

Optimizations improve scalability of DC-based query systems.

Experimental results validate the effectiveness of the proposed methods.

Abstract

Differential computation (DC) is a highly general incremental computation/view maintenance technique that can maintain the output of an arbitrary and possibly recursive dataflow computation upon changes to its base inputs. As such, it is a promising technique for graph database management systems (GDBMS) that support continuous recursive queries over dynamic graphs. Although differential computation can be highly efficient for maintaining these queries, it can require a prohibitively large amount of memory. This paper studies how to reduce the memory overhead of DC with the goal of increasing the scalability of systems that adopt it. We propose a suite of optimizations that are based on dropping the differences of operators, both completely or partially, and recomputing these differences when necessary. We propose deterministic and probabilistic data structures to keep track of the dropped differences. Extensive experiments demonstrate that the optimizations can improve the scalability of a DC-based continuous query processor.