Learned Query Optimizer in Alibaba MaxCompute: Challenges, Analysis, and Solutions
Lianggui Weng, Dandan Liu, Wenzhuang Zhu, Rong Zhu, Junzheng Zheng, Bolin Ding, Zhiguo Zhang, Jingren Zhou

TL;DR
This paper presents LOAM, a deployment-oriented learned query optimizer for Alibaba MaxCompute, addressing real-world challenges with a novel, system-agnostic design that improves cost estimation and reduces CPU usage.
Contribution
The paper introduces LOAM, a new learned query optimization framework that overcomes deployment challenges and generalizes to production environments without traditional model refinement.
Findings
LOAM achieves up to 30% CPU cost savings on production workloads.
It introduces a statistics-free plan encoding leveraging operator semantics.
LOAM provides theoretical bounds and practical strategies for environment impact smoothing.
Abstract
Existing learned query optimizers remain ill-suited to modern distributed, multi-tenant data warehouses due to idealized modeling assumptions and design choices. Using Alibaba's MaxCompute as a representative, we surface four fundamental, system-agnostic challenges for any deployable learned query optimizer: 1) highly dynamic execution environments that induce large variance in plan costs; 2) potential absence of input statistics needed for cost estimation; 3) infeasibility of conventional model refinement; and 4) uncertain benefits across different workloads. These challenges expose a deep mismatch between theoretical advances and production realities and demand a principled, deployment-first redesign of learned optimizers. To bridge this gap, we present LOAM, a one-stop learned query optimization framework for MaxCompute. Its design principles and techniques generalize and are…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Database Systems and Queries · Graph Theory and Algorithms · Data Management and Algorithms
