Partout: A Distributed Engine for Efficient RDF Processing

TL;DR

Partout is a distributed RDF processing engine that efficiently manages large-scale semantic data by optimized data fragmentation, distribution, and query planning, outperforming existing systems.

Contribution

It introduces a novel distributed engine with query log-based RDF fragmentation, optimized data placement, and efficient query execution for large-scale semantic web data.

Findings

Outperforms state-of-the-art RDF processing systems

Efficient handling of updates in distributed RDF data

Produces optimized query execution plans for ad-hoc SPARQL queries

Abstract

The increasing interest in Semantic Web technologies has led not only to a rapid growth of semantic data on the Web but also to an increasing number of backend applications with already more than a trillion triples in some cases. Confronted with such huge amounts of data and the future growth, existing state-of-the-art systems for storing RDF and processing SPARQL queries are no longer sufficient. In this paper, we introduce Partout, a distributed engine for efficient RDF processing in a cluster of machines. We propose an effective approach for fragmenting RDF data sets based on a query log, allocating the fragments to nodes in a cluster, and finding the optimal configuration. Partout can efficiently handle updates and its query optimizer produces efficient query execution plans for ad-hoc SPARQL queries. Our experiments show the superiority of our approach to state-of-the-art approaches for partitioning and distributed SPARQL query processing.