A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation

TL;DR

This paper introduces UAE, a unified deep autoregressive model that learns joint data distributions for cardinality estimation by leveraging both data and query workload, improving accuracy and efficiency.

Contribution

The paper proposes UAE, a novel model that integrates data-driven and query-driven learning for cardinality estimation using a differentiable sampling technique.

Findings

UAE achieves single-digit multiplicative error at tail.

UAE outperforms state-of-the-art methods in accuracy.

UAE is both space and time efficient.

Abstract

Cardinality estimation is a fundamental problem in database systems. To capture the rich joint data distributions of a relational table, most of the existing work either uses data as unsupervised information or uses query workload as supervised information. Very little work has been done to use both types of information, and cannot fully make use of both types of information to learn the joint data distribution. In this work, we aim to close the gap between data-driven and query-driven methods by proposing a new unified deep autoregressive model, UAE, that learns the joint data distribution from both the data and query workload. First, to enable using the supervised query information in the deep autoregressive model, we develop differentiable progressive sampling using the Gumbel-Softmax trick. Second, UAE is able to utilize both types of information to learn the joint data distribution in a single model. Comprehensive experimental results demonstrate that UAE achieves single-digit multiplicative error at tail, better accuracies over state-of-the-art methods, and is both space and time efficient.