On the Effectiveness of Pretrained Models for API Learning

TL;DR

This paper evaluates the effectiveness of pre-trained Transformer models for automatic API sequence generation from natural language queries, showing they outperform previous methods by around 11% on a large GitHub dataset.

Contribution

It is the first to systematically assess pre-trained Transformer models for API learning, demonstrating their superior performance and exploring tokenization strategies to enhance results.

Findings

PTMs generate more accurate API sequences than previous methods.

PTMs outperform existing approaches by approximately 11%.

Tokenization approaches significantly boost PTMs' performance.

Abstract

Developers frequently use APIs to implement certain functionalities, such as parsing Excel Files, reading and writing text files line by line, etc. Developers can greatly benefit from automatic API usage sequence generation based on natural language queries for building applications in a faster and cleaner manner. Existing approaches utilize information retrieval models to search for matching API sequences given a query or use RNN-based encoder-decoder to generate API sequences. As it stands, the first approach treats queries and API names as bags of words. It lacks deep comprehension of the semantics of the queries. The latter approach adapts a neural language model to encode a user query into a fixed-length context vector and generate API sequences from the context vector. We want to understand the effectiveness of recent Pre-trained Transformer based Models (PTMs) for the API learning task. These PTMs are trained on large natural language corpora in an unsupervised manner to retain contextual knowledge about the language and have found success in solving similar Natural Language Processing (NLP) problems. However, the applicability of PTMs has not yet been explored for the API sequence generation task. We use a dataset that contains 7 million annotations collected from GitHub to evaluate the PTMs empirically. This dataset was also used to assess previous approaches. Based on our results, PTMs generate more accurate API sequences and outperform other related methods by around 11%. We have also identified two different tokenization approaches that can contribute to a significant boost in PTMs' performance for the API sequence generation task.