Enhancing the Capabilities of Large Language Models for API calls through Knowledge Graphs

TL;DR

This paper presents KG2data, a system that enhances large language models' ability to perform API calls in meteorology by integrating knowledge graphs, improving accuracy, reasoning, and data retrieval without extensive fine-tuning.

Contribution

The paper introduces KG2data, a novel framework combining knowledge graphs with LLMs and tool-use techniques to improve API call accuracy and domain-specific reasoning in knowledge-intensive fields.

Findings

KG2data achieves 88.57% call correctness, outperforming baselines.

Using knowledge graphs improves content retrieval and complex query handling.

The system reduces fine-tuning costs and adapts to evolving domain knowledge.

Abstract

API calls by large language models (LLMs) offer a cutting-edge approach for data analysis. However, their ability to effectively utilize tools via API calls remains underexplored in knowledge-intensive domains like meteorology. This paper introduces KG2data, a system that integrates knowledge graphs, LLMs, ReAct agents, and tool-use technologies to enable intelligent data acquisition and query handling in the meteorological field. Using a virtual API, we evaluate API call accuracy across three metrics: name recognition failure, hallucination failure, and call correctness. KG2data achieves superior performance (1.43%, 0%, 88.57%) compared to RAG2data (16%, 10%, 72.14%) and chat2data (7.14%, 8.57%, 71.43%). KG2data differs from typical LLM-based systems by addressing their limited access to domain-specific knowledge, which hampers performance on complex or terminology-rich queries. By using a knowledge graph as persistent memory, our system enhances content retrieval, complex query handling, domain-specific reasoning, semantic relationship resolution, and heterogeneous data integration. It also mitigates the high cost of fine-tuning LLMs, making the system more adaptable to evolving domain knowledge and API structures. In summary, KG2data provides a novel solution for intelligent, knowledge-based question answering and data analysis in domains with high knowledge demands.