Enabling Incremental Query Re-Optimization

TL;DR

This paper introduces an incremental query re-optimization approach that allows adaptive, cost-based query plan updates in real-time environments, improving responsiveness over traditional non-incremental methods.

Contribution

It presents a novel incremental re-optimization architecture based on recursive datalog query enumeration, enabling faster adaptation in streaming and repeated query scenarios.

Findings

Supports cost-based initial optimization and frequent adaptivity

Effective pruning strategies for static and incremental cases

Implemented within an existing system demonstrating practical benefits

Abstract

As declarative query processing techniques expand in scope --- to the Web, data streams, network routers, and cloud platforms --- there is an increasing need for adaptive query processing techniques that can re-plan in the presence of failures or unanticipated performance changes. A status update on the data distributions or the compute nodes may have significant repercussions on the choice of which query plan should be running. Ideally, new system architectures would be able to make cost-based decisions about reallocating work, migrating data, etc., and react quickly as real-time status information becomes available. Existing cost-based query optimizers are not incremental in nature, and must be run "from scratch" upon each status or cost update. Hence, they generally result in adaptive schemes that can only react slowly to updates. An open question has been whether it is possible to build a cost-based re-optimization architecture for adaptive query processing in a streaming or repeated query execution environment, e.g., by incrementally updating optimizer state given new cost information. We show that this can be achieved beneficially, especially for stream processing workloads. Our techniques build upon the recently proposed approach of formulating query plan enumeration as a set of recursive datalog queries; we develop a variety of novel optimization approaches to ensure effective pruning in both static and incremental cases. We implement our solution within an existing research query processing system, and show that it effectively supports cost-based initial optimization as well as frequent adaptivity.