Evaluating Pre-trained Language Models for Repairing API Misuses

TL;DR

This paper empirically evaluates the effectiveness of pre-trained language models in repairing API misuses, demonstrating that PLMs outperform traditional APR tools in this task.

Contribution

It provides the first comprehensive study on applying 11 learning-aided APR tools, including 9 state-of-the-art PLMs, to API misuse repair, filling a significant research gap.

Findings

PLMs outperform traditional APR tools in API misuse repair

CodeT5 achieves the highest exact match accuracy among tested models

The study offers insights and future directions for PLM-based API repair

Abstract

API misuses often lead to software bugs, crashes, and vulnerabilities. While several API misuse detectors have been proposed, there are no automatic repair tools specifically designed for this purpose. In a recent study, test-suite-based automatic program repair (APR) tools were found to be ineffective in repairing API misuses. Still, since the study focused on non-learning-aided APR tools, it remains unknown whether learning-aided APR tools are capable of fixing API misuses. In recent years, pre-trained language models (PLMs) have succeeded greatly in many natural language processing tasks. There is a rising interest in applying PLMs to APR. However, there has not been any study that investigates the effectiveness of PLMs in repairing API misuse. To fill this gap, we conduct a comprehensive empirical study on 11 learning-aided APR tools, which include 9 of the state-of-the-art general-purpose PLMs and two APR tools. We evaluate these models with an API-misuse repair dataset, consisting of two variants. Our results show that PLMs perform better than the studied APR tools in repairing API misuses. Among the 9 pre-trained models tested, CodeT5 is the best performer in the exact match. We also offer insights and potential exploration directions for future research.