HDTCat: let's make HDT scale

TL;DR

HDTCat is a new algorithm and tool that efficiently joins HDT files with minimal memory usage, enabling scalable processing of large RDF datasets.

Contribution

We introduce HDTCat, a low-memory algorithm and command line tool for joining HDT files, facilitating scalable HDT generation from large RDF datasets.

Findings

HDTCat can join HDT files with low memory consumption.

It enables divide-and-conquer processing of large RDF datasets.

The tool improves scalability of HDT serialization.

Abstract

HDT (Header, Dictionary, Triples) is a serialization for RDF. HDT has become very popular in the last years because it allows to store RDF data with a small disk footprint, while remaining at the same time queriable. For this reason HDT is often used when scalability becomes an issue. Once RDF data is serialized into HDT, the disk footprint to store it and the memory footprint to query it are very low. However, generating HDT files from raw text RDF serializations (like N-Triples) is a time-consuming and (especially) memory-consuming task. In this publication we present HDTCat, an algorithm and command line tool to join two HDT files with low memory footprint. HDTCat can be used in a divide-and-conquer strategy to generate HDT files from huge datasets using a low-memory footprint.