Learned Query Optimizer in Alibaba MaxCompute: Challenges, Analysis, and Solutions

TL;DR

This paper presents LOAM, a deployment-oriented learned query optimizer for Alibaba MaxCompute, addressing real-world challenges with a novel, system-agnostic design that improves cost estimation and reduces CPU usage.

Contribution

The paper introduces LOAM, a new learned query optimization framework that overcomes deployment challenges and generalizes to production environments without traditional model refinement.

Findings

LOAM achieves up to 30% CPU cost savings on production workloads.

It introduces a statistics-free plan encoding leveraging operator semantics.

LOAM provides theoretical bounds and practical strategies for environment impact smoothing.

Abstract

Existing learned query optimizers remain ill-suited to modern distributed, multi-tenant data warehouses due to idealized modeling assumptions and design choices. Using Alibaba's MaxCompute as a representative, we surface four fundamental, system-agnostic challenges for any deployable learned query optimizer: 1) highly dynamic execution environments that induce large variance in plan costs; 2) potential absence of input statistics needed for cost estimation; 3) infeasibility of conventional model refinement; and 4) uncertain benefits across different workloads. These challenges expose a deep mismatch between theoretical advances and production realities and demand a principled, deployment-first redesign of learned optimizers. To bridge this gap, we present LOAM, a one-stop learned query optimization framework for MaxCompute. Its design principles and techniques generalize and are readily adaptable to similar systems. Architecturally, LOAM introduces a statistics-free plan encoding that leverages operator semantics and historical executions to infer details about data distributions and explicitly encodes the execution environments of training queries to learn their impacts on plan costs. For online queries with unknown environments at prediction time, LOAM provides a theoretical bound on the achievable performance and a practical strategy to smooth the environmental impacts on cost estimations. For system operating, LOAM integrates domain adaptation techniques into training to generalize effectively to online query plans without requiring conventional refinement. Additionally, LOAM includes a lightweight project selector to prioritize high-benefit deployment projects. LOAM has seen up to 30% CPU cost savings over MaxCompute's native query optimizer on production workloads, which could translate to substantial real-world resource savings.