An Experience Report of Large Scale Federations

TL;DR

This paper reports on an extensive experimental study of large-scale RDF federations using Bio2RDF data sources, evaluating performance, design choices, and challenges in both local and hybrid network settings.

Contribution

It provides practical insights, technical details, and an evaluation of federated semantic data management at a large scale, highlighting bottlenecks and future research directions.

Findings

Federated semantic data management is feasible at large scale.

Network latency impacts federation performance significantly.

Identifies bottlenecks and opportunities for optimization.

Abstract

We present an experimental study of large-scale RDF federations on top of the Bio2RDF data sources, involving 29 data sets with more than four billion RDF triples deployed in a local federation. Our federation is driven by FedX, a highly optimized federation mediator for Linked Data. We discuss design decisions, technical aspects, and experiences made in setting up and optimizing the Bio2RDF federation, and present an exhaustive experimental evaluation of the federation scenario. In addition to a controlled setting with local federation members, we study implications arising in a hybrid setting, where local federation members interact with remote federation members exhibiting higher network latency. The outcome demonstrates the feasibility of federated semantic data management in general and indicates remaining bottlenecks and research opportunities that shall serve as a guideline for future work in the area of federated semantic data processing.