CardOOD: Robust Query-driven Cardinality Estimation under Out-of-Distribution

TL;DR

CardOOD introduces a robust framework for query-driven cardinality estimation that effectively mitigates out-of-distribution issues, enhancing accuracy and reliability in query optimization tasks.

Contribution

The paper presents a novel learning framework that extends transfer and robust learning techniques, including a new self-supervised algorithm tailored for cardinality estimation, to improve OOD robustness.

Findings

Effective mitigation of OOD problems in cardinality estimation.

Integration of CardOOD into PostgreSQL improves query optimization.

New self-supervised learning algorithm models cardinality constraints.

Abstract

Query-driven learned estimators are accurate, flexible, and lightweight alternatives to traditional estimators in query optimization. However, existing query-driven approaches struggle with the Out-of-distribution (OOD) problem, where the test workload distribution differs from the training workload, leading to performancedegradation. In this paper, we present CardOOD, a general learning framework designed to construct robust query-driven cardinality estimators that are resilient against the OOD problem. Our framework focuses on offline training algorithms that develop one-off models from a static workload, suitable for model initialization and periodic retraining. In CardOOD, we extend classical transfer/robust learning techniques to train query-driven cardinalityestimators, and the algorithms fall into three categories: representation learning, data manipulation, and new learning strategies. As these learning techniques are originally evaluated in computervision tasks, we also propose a new learning algorithm that exploits the property of cardinality estimation. This algorithm, lying in the category of new learning strategy, models the partial order constraint of cardinalities by a self-supervised learning task. Comprehensive experimental studies demonstrate the efficacy of the algorithms of CardOOD in mitigating the OOD problem to varying extents. We further integrate CardOOD into PostgreSQL, showcasing its practical utility in query optimization.