PIPE-RDF: An LLM-Assisted Pipeline for Enterprise RDF Benchmarking

TL;DR

PIPE-RDF is a pipeline that creates realistic, schema-specific RDF benchmarks for enterprise knowledge graphs, enabling better evaluation of natural language interfaces and question-answering systems.

Contribution

It introduces a novel pipeline that generates balanced, schema-specific RDF benchmarks with high validity, tailored for enterprise knowledge graph evaluation.

Findings

Achieved 100% validity after repair in benchmark generation

Generated 450 question-SPARQL pairs across nine categories

Demonstrated pipeline's effectiveness on a large enterprise RDF dataset

Abstract

Enterprises rely on RDF knowledge graphs and SPARQL to expose operational data through natural language interfaces, yet public KGQA benchmarks do not reflect proprietary schemas, prefixes, or query distributions. We present PIPE-RDF, a three-phase pipeline that constructs schema-specific NL-SPARQL benchmarks using reverse querying, category-balanced template generation, retrieval-augmented prompting, deduplication, and execution-based validation with repair. We instantiate PIPE-RDF on a fixed-schema company-location slice (5,000 companies) derived from public RDF data and generate a balanced benchmark of 450 question-SPARQL pairs across nine categories. The pipeline achieves 100% parse and execution validity after repair, with pre-repair validity rates of 96.5%-100% across phases. We report entity diversity metrics, template coverage analysis, and cost breakdowns to support deployment planning. We release structured artifacts (CSV/JSONL, logs, figures) and operational metrics to support model evaluation and system planning in real-world settings. Code is available at https://github.com/suraj-ranganath/PIPE-RDF.