KG2QA: Knowledge Graph-enhanced Retrieval-augmented Generation for Communication Standards Question Answering

TL;DR

KG2QA is a novel framework that combines large language models with a domain-specific knowledge graph to improve question answering accuracy and factual grounding for communication standards.

Contribution

The paper introduces KG2QA, integrating a knowledge graph with fine-tuned LLMs via RAG, and constructs a new dataset and knowledge graph for communication standards QA.

Findings

Significant BLEU-4 score improvement from 18.86 to 66.90.

Enhanced factual accuracy and relevance demonstrated by evaluation.

Open-sourced code and data for community use.

Abstract

The rapid evolution of communication technologies has led to an explosion of standards, rendering traditional expert-dependent consultation methods inefficient and slow. To address this challenge, we propose \textbf{KG2QA}, a question answering (QA) framework for communication standards that integrates fine-tuned large language models (LLMs) with a domain-specific knowledge graph (KG) via a retrieval-augmented generation (RAG) pipeline. We construct a high-quality dataset of 6,587 QA pairs from ITU-T recommendations and fine-tune Qwen2.5-7B-Instruct, achieving significant performance gains: BLEU-4 increases from 18.86 to 66.90, outperforming both the base model and Llama-3-8B-Instruct. A structured KG containing 13,906 entities and 13,524 relations is built using LLM-assisted triple extraction based on a custom ontology. In our KG-RAG pipeline, the fine-tuned LLMs first retrieves relevant knowledge from KG, enabling more accurate and factually grounded responses. Evaluated by DeepSeek-V3 as a judge, the KG-enhanced system improves performance across five dimensions, with an average score increase of 2.26\%, demonstrating superior factual accuracy and relevance. Integrated with Web platform and API, KG2QA delivers an efficient and interactive user experience. Our code and data have been open-sourced https://github.com/luozhongze/KG2QA.