An Empirical Evaluation of Cost-based Federated SPARQL Query Processing Engines

TL;DR

This paper introduces new evaluation metrics for assessing the accuracy of cardinality estimators in cost-based federated SPARQL query engines, providing detailed insights into their performance and impact on query runtime.

Contribution

It presents novel evaluation metrics and a comprehensive benchmarking approach for analyzing the accuracy of cardinality estimators in federated SPARQL engines.

Findings

New metrics enable fine-grained benchmarking

Evaluation reveals the impact of estimation errors on performance

Provides insights for future engine development

Abstract

Finding a good query plan is key to the optimization of query runtime. This holds in particular for cost-based federation engines, which make use of cardinality estimations to achieve this goal. A number of studies compare SPARQL federation engines across different performance metrics, including query runtime, result set completeness and correctness, number of sources selected and number of requests sent. Albeit informative, these metrics are generic and unable to quantify and evaluate the accuracy of the cardinality estimators of cost-based federation engines. To thoroughly evaluate cost-based federation engines, the effect of estimated cardinality errors on the overall query runtime performance must be measured. In this paper, we address this challenge by presenting novel evaluation metrics targeted at a fine-grained benchmarking of cost-based federated SPARQL query engines. We evaluate five cost-based federated SPARQL query engines using existing as well as novel evaluation metrics by using LargeRDFBench queries. Our results provide a detailed analysis of the experimental outcomes that reveal novel insights, useful for the development of future cost-based federated SPARQL query processing engines.