Let's Chat to Find the APIs: Connecting Human, LLM and Knowledge Graph through AI Chain

TL;DR

This paper introduces a knowledge-guided query clarification method using LLMs and knowledge graphs to improve API recommendation accuracy, efficiency, and fluency, overcoming limitations of previous KG-based approaches.

Contribution

It proposes a novel AI chain framework that integrates LLMs with knowledge graphs for enhanced API query clarification, addressing OOV issues and noise filtering.

Findings

Significant improvement in MRR over baselines, up to 63.9%.

High scores (close to 5) for each unit in the AI chain.

Knowledge guidance and pathfinding are crucial, boosting MAP by over 20%.

Abstract

API recommendation methods have evolved from literal and semantic keyword matching to query expansion and query clarification. The latest query clarification method is knowledge graph (KG)-based, but limitations include out-of-vocabulary (OOV) failures and rigid question templates. To address these limitations, we propose a novel knowledge-guided query clarification approach for API recommendation that leverages a large language model (LLM) guided by KG. We utilize the LLM as a neural knowledge base to overcome OOV failures, generating fluent and appropriate clarification questions and options. We also leverage the structured API knowledge and entity relationships stored in the KG to filter out noise, and transfer the optimal clarification path from KG to the LLM, increasing the efficiency of the clarification process. Our approach is designed as an AI chain that consists of five steps, each handled by a separate LLM call, to improve accuracy, efficiency, and fluency for query clarification in API recommendation. We verify the usefulness of each unit in our AI chain, which all received high scores close to a perfect 5. When compared to the baselines, our approach shows a significant improvement in MRR, with a maximum increase of 63.9% higher when the query statement is covered in KG and 37.2% when it is not. Ablation experiments reveal that the guidance of knowledge in the KG and the knowledge-guided pathfinding strategy are crucial for our approach's performance, resulting in a 19.0% and 22.2% increase in MAP, respectively. Our approach demonstrates a way to bridge the gap between KG and LLM, effectively compensating for the strengths and weaknesses of both.