Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We?

TL;DR

This paper revisits API recommendation, benchmarking 11 approaches and 4 IDEs, and introduces APIBench to evaluate and guide future improvements in API recommendation systems.

Contribution

It provides a unified benchmark for API recommendation, categorizes approaches, and offers insights and directions for enhancing API recommendation performance.

Findings

APIBench enables standardized evaluation of API recommendation methods.

Different approaches vary significantly in performance and applicability.

Insights highlight data, query reformulation, and cross-domain challenges.

Abstract

Application Programming Interfaces (APIs), which encapsulate the implementation of specific functions as interfaces, greatly improve the efficiency of modern software development. As numbers of APIs spring up nowadays, developers can hardly be familiar with all the APIs, and usually need to search for appropriate APIs for usage. So lots of efforts have been devoted to improving the API recommendation task. However, it has been increasingly difficult to gauge the performance of new models due to the lack of a uniform definition of the task and a standardized benchmark. For example, some studies regard the task as a code completion problem; while others recommend relative APIs given natural language queries. To reduce the challenges and better facilitate future research, in this paper, we revisit the API recommendation task and aim at benchmarking the approaches. Specifically, the paper groups the approaches into two categories according to the task definition, i.e., query-based API recommendation and code-based API recommendation. We study 11 recently-proposed approaches along with 4 widely-used IDEs. One benchmark named as APIBench is then built for the two respective categories of approaches. Based on APIBench, we distill some actionable insights and challenges for API recommendation. We also achieve some implications and directions for improving the performance of recommending APIs, including data source selection, appropriate query reformulation, low resource setting, and cross-domain adaptation.