Not All RDF is Created Equal: Investigating RDF Load Times on Resource-Constrained Devices

TL;DR

This paper evaluates how RDF load times vary across different datasets and hardware, revealing significant differences and emphasizing the need for diverse benchmarks in resource-constrained environments.

Contribution

It introduces the concept of relative loading speed (RLS) and provides an experimental analysis of RDF load times across multiple datasets and devices, highlighting dataset-dependent performance.

Findings

Loading speed varies by up to a factor of 9.01 between datasets.

Using multiple datasets is crucial for accurate RDF store evaluation.

Published data and code for reproducibility.

Abstract

As the role of knowledge-based systems in IoT keeps growing, ensuring resource efficiency of RDF stores becomes critical. However, up until now benchmarks of RDF stores were most often conducted with only one dataset, and the differences between the datasets were not explored in detail. In this paper, our objective is to close this research gap by experimentally evaluating the load times of eight diverse RDF datasets from the RiverBench benchmark suite. In the experiments, we use five different RDF store implementations and several resource-constrained hardware platforms. To analyze the results, we introduce the notion of relative loading speed (RLS), allowing us to observe that the loading speed can differ between datasets by as much as a factor of 9.01. This serves as clear evidence that ``not all RDF is created equal'' and stresses the importance of using multiple benchmark datasets in evaluations. We outline the possible reasons for this drastic difference, which should be further investigated in future work. To this end, we published the data, code, and the results of our experiments.