Performance Evaluation of LLMs in Automated RDF Knowledge Graph Generation

TL;DR

This study systematically evaluates various LLMs and prompting strategies for automating RDF knowledge graph generation from complex cloud logs, demonstrating high accuracy with Few-Shot learning especially using Llama.

Contribution

It introduces a controlled framework and a new Log-to-KG dataset for objective benchmarking of LLMs in RDF extraction from cloud logs.

Findings

Llama achieves 99.35% F1 score and 100% valid RDF output with Few-Shot prompting.

Few-Shot learning outperforms other prompting strategies across multiple LLM architectures.

Prompt design and contextual examples are crucial for accurate RDF extraction.

Abstract

Cloud systems generate large, heterogeneous log data containing critical infrastructure, application, and security information. Transforming these logs into RDF triples enables their integration into knowledge graphs, improving interpretability, root-cause analysis, and cross-service reasoning beyond what raw logs allow. Large Language Models (LLMs) offer a promising approach to automate RDF knowledge graph generation; however, their effectiveness on complex cloud logs remains largely unexplored. In this paper, we evaluate multiple LLM architectures and prompting strategies for automated RDF extraction using a controlled framework with two pipelines for systematically processing semi-structured log data. The extraction pipeline integrates multiple LLMs to identify relevant entities and relationships, automatically generating subject-predicate-object triples. These outputs are evaluated using a dedicated validation pipeline with both syntactic and semantic metrics to assess accuracy, completeness, and quality. Due to the lack of public ground-truth datasets, we created a reference Log-to-KG dataset from OpenStack logs using manual annotation and ontology-driven methods, enabling objective baseline. Our analysis shows that Few-Shot learning is the most effective strategy, with Llama achieving a 99.35% F1 score and 100% valid RDF output while Qwen, NuExtract, and Gemma also perform well under Few-Shot prompting, with Chain-of-Thought approaches maintaining similar accuracy. One-Shot prompting offers a lighter but effective alternative, while Zero-Shot and advanced strategies such as Tree-of-Thought, Self-Critique, and Generate-Multiple perform substantially worse. These results highlight the importance of contextual examples and prompt design for accurate RDF extraction and reveal model-specific limitations across LLM architectures.