GenJoin: Conditional Generative Plan-to-Plan Query Optimizer that Learns from Subplan Hints

TL;DR

GenJoin is a novel generative query optimizer that learns from subplan hints to produce superior query plans, outperforming traditional and state-of-the-art learned optimizers on real-world benchmarks.

Contribution

It introduces a generative approach to query optimization that leverages subplan hints, reducing reliance on extensive training data and improving plan quality.

Findings

GenJoin outperforms PostgreSQL on benchmark tests.

It surpasses existing learned optimizers in various workloads.

The approach demonstrates stable and consistent improvements.

Abstract

Query optimization has become a research area where classical algorithms are being challenged by machine learning algorithms. At the same time, recent trends in learned query optimizers have shown that it is prudent to take advantage of decades of database research and augment classical query optimizers by shrinking the plan search space through different types of hints (e.g. by specifying the join type, scan type or the order of joins) rather than completely replacing the classical query optimizer with machine learning models. It is especially relevant for cases when classical optimizers cannot fully enumerate all logical and physical plans and, as an alternative, need to rely on less robust approaches like genetic algorithms. However, even symbiotically learned query optimizers are hampered by the need for vast amounts of training data, slow plan generation during inference and unstable results across various workload conditions. In this paper, we present GenJoin - a novel learned query optimizer that considers the query optimization problem as a generative task and is capable of learning from a random set of subplan hints to produce query plans that outperform the classical optimizer. GenJoin is the first learned query optimizer that significantly and consistently outperforms PostgreSQL as well as state-of-the-art methods on two well-known real-world benchmarks across a variety of workloads using rigorous machine learning evaluations.