CONCERTO: Complex Query Execution Mechanism-Aware Learned Cost Estimation

TL;DR

CONCERTO introduces a novel learned cost estimation framework for complex query execution mechanisms, effectively modeling resource competition and parallelism to improve prediction accuracy in modern DBMSs.

Contribution

It develops a new approach combining GATs and TCNs to accurately predict query performance for complex, parallel, and resource-sharing execution mechanisms.

Findings

CONCERTO outperforms existing prediction methods in accuracy.

It effectively models resource competition among concurrent operators.

The approach adapts to complex execution mechanisms beyond traditional plans.

Abstract

With the growing demand for massive data analysis, many DBMSs have adopted complex underlying query execution mechanisms, including vectorized operators, parallel execution, and dynamic pipeline modifications. However, there remains a lack of targeted Query Performance Prediction (QPP) methods for these complex execution mechanisms and their interactions, as most existing approaches focus on traditional tree-shaped query plans and static serial executors. To address this challenge, this paper proposes CONCERTO, a Complex query executiON meChanism-awaE leaRned cosT estimatiOn method. CONCERTO first establishes independent resource cost models for each physical operator. It then constructs a Directed Acyclic Graph (DAG) consisting of a dataflow tree backbone and resource competition relationships among concurrent operators. After calibrating the cost impact of parallel operator execution using Graph Attention Networks (GATs) with additional attention mechanisms, CONCERTO extracts and aggregates cost vector trees through Temporal Convolutional Networks (TCNs), ultimately achieving effective query performance prediction. Experimental results demonstrate that CONCERTO achieves higher prediction accuracy than existing methods.