Robust Plan Evaluation based on Approximate Probabilistic Machine Learning

TL;DR

This paper introduces Roq, a risk-aware machine learning framework for robust query plan evaluation in RDBMSs, significantly improving plan robustness despite inaccurate estimates.

Contribution

It presents a novel formalization of robustness in query optimization and a probabilistic ML-based approach for plan evaluation and selection.

Findings

Roq outperforms existing methods in robustness.

The learned cost model accurately predicts execution costs and risks.

Significant improvements in query plan robustness are demonstrated.

Abstract

Query optimizers in RDBMSs search for execution plans expected to be optimal for given queries. They use parameter estimates, often inaccurate, and make assumptions that may not hold in practice. Consequently, they may select plans that are suboptimal at runtime if estimates and assumptions are not valid. Therefore, they do not sufficiently support robust query optimization. Using ML to improve data systems has shown promising results for query optimization. Inspired by this, we propose Robust Query Optimizer (Roq), a holistic framework based on a risk-aware learning approach. Roq includes a novel formalization of the notion of robustness in the context of query optimization and a principled approach for its quantification and measurement based on approximate probabilistic ML. It also includes novel strategies and algorithms for query plan evaluation and selection. Roq includes a novel learned cost model that is designed to predict the cost of query execution and the associated risks and performs query optimization accordingly. We demonstrate that Roq provides significant improvements in robust query optimization compared with the state-of-the-art.