Q-ARE: An Evaluation Dataset for Query Based API Recommendation

TL;DR

Q-ARE is a new dataset and benchmark for evaluating the semantic understanding of query-based API recommendation methods, highlighting challenges with multi-level invocation structures.

Contribution

The paper introduces Q-ARE, a comprehensive dataset with novel metrics for assessing API recommendation methods on open-source Java projects.

Findings

Performance drops with increased API Call Depth.

Existing methods struggle with multi-level invocation chains.

Q-ARE provides a new benchmark for future API recommendation research.

Abstract

As software systems grow in scale, developers face increasing difficulty in selecting appropriate Application Programming Interfaces (APIs) from numerous options. Efficiently identifying APIs that satisfy functional requirements has become a key challenge. To evaluate the semantic understanding of existing query-based API recommendation methods, this paper constructs Q-ARE (Query-based API Recommendation Evaluation), a dataset based on open-source Java projects from GitHub. Methods and their invocation chains are analyzed to identify third-party APIs directly or indirectly invoked by target methods, recursively expanding multi-level invocations to unify hierarchical call structures into API recommendation target sets. Furthermore, we introduce two metrics: API Call Depth, measuring the invocation distance between a query method and a target API, and Invocation Density, quantifying the proportion of code lines associated with the target API in the invocation chain. Based on Q-ARE, we systematically evaluate several query-based API recommendation methods and general Large Language Models (LLMs). Results show that performance drops significantly as API Call Depth increases and invocation density decreases, indicating that existing methods still struggle with multi-level method invocation structures. Q-ARE and its metrics provide a new benchmark for assessing semantic understanding in API recommendation and offer insights for improving future algorithms.