The Case for Cardinality Lower Bounds

TL;DR

This paper introduces xBound, a theoretical framework for computing provable join size lower bounds, significantly reducing underestimation errors in cardinality estimation and improving query performance in production systems.

Contribution

The paper presents xBound, the first method for provable join size lower bounds, addressing a critical gap in cardinality estimation for database systems.

Findings

xBound corrects 23.6% of underestimates in Fabric DW.

Query speedups of up to 20.1x achieved with xBound.

Demonstrates practical benefits of provable lower bounds in production environments.

Abstract

Despite decades of research, cardinality estimation remains the optimizer's Achilles heel, with industrial-strength systems exhibiting a systemic tendency toward underestimation. At cloud scale, this is a severe production vulnerability: in Microsoft's Fabric Data Warehouse (DW), a mere 0.05% of extreme underestimates account for 95% of all CPU under-allocation, causing preventable slowdowns for thousands of queries daily. Yet recent theoretical work on provable upper bounds only corrects overestimation, leaving the more harmful problem of underestimation unaddressed. We argue that closing this gap is an urgent priority for the database community. As a vital step toward this goal, we introduce xBound, the first theoretical framework for computing provable join size lower bounds. By clipping the optimizer's estimates from below, xBound offers strict mathematical safety nets demanded by production systems - using only a handful of lightweight base table statistics. We demonstrate xBound's practical impact on Fabric DW: on the StackOverflow-CEB benchmark, it corrects 23.6% of Fabric DW's underestimates, yielding end-to-end query speedups of up to 20.1x, demonstrating that even a first step toward provable lower bounds can deliver meaningful production gains and motivating the community to further pursue this critical, open direction.