Query Optimization in the Wild: Realities and Trends

TL;DR

This paper reviews the evolution of query optimization in industry, highlighting current challenges and emerging trends such as feedback loops, workload optimization, and modular architectures that aim to enhance flexibility and performance.

Contribution

It provides an industrial perspective on the limitations of traditional query optimizers and identifies three key industry trends shaping future developments.

Findings

Enhanced robustness through tighter feedback loops

Expansion from single-query to workload optimization

Shift towards modular, composable architectures

Abstract

For nearly half a century, the core design of query optimizers in industrial database systems has remained remarkably stable, relying on foundational principles from System R and the Volcano/Cascades framework. However, the rise of cloud computing, massive data volumes, and unified data platforms has exposed the limitations of this traditional, monolithic architecture. Taking an industrial perspective, this paper reviews the past and present of query optimization in production systems and identifies the challenges they face today. Then this paper highlights three key trends gaining momentum in the industry that promise to address these challenges. First, a tighter feedback loop between query optimization and query execution is being used to improve the robustness of query performance. Second, the scope of optimization is expanding from a single query to entire workloads through the convergence of query optimization and workload optimization. Third, and perhaps most transformatively, the industry is moving from monolithic designs to composable architectures that foster agility and cross-engine collaboration. Together, these trends chart a clear path toward a more dynamic, holistic, and adaptable future for query optimization in practice.