Tail-aware N-version Machine Learning Models for Reliable API Recommendation

TL;DR

This paper introduces NvRec, a multi-model API recommendation system that improves reliability by filtering out tail API suggestions, demonstrating high true accept rates and balanced rejection rates on Java datasets.

Contribution

The paper proposes NvRec, a novel multi-model framework that enhances API recommendation reliability by profiling and filtering tail API outputs during inference.

Findings

Five-model NvRec achieves 83.1% true accept rate.

Three-model NvRec reaches 83.8% true accept rate with high reliability.

NvRec balances true accept and rejection rates effectively.

Abstract

Machine learning (ML)-based API recommendation helps developers efficiently identify suitable APIs to complement the application code. However, code datasets used to train ML models often exhibit a long-tail distribution, leading to unreliable API recommendations, especially for infrequently used API methods at the tail of the distribution. To address this issue, we propose N-version API Recommendation (NvRec), which leverages N different versions of ML models to enhance the reliability of API sequence recommendations by suppressing unreliable outputs entailing tail APIs. NvRec leverages a set of available ML models and profiles their performance on individual API methods with their tail properties. The generated model profile is used at inference time to filter out unreliable API recommendations and determine the final output. We implement NvRec using five API recommendation models, including CodeBERT, CodeT5, MulaRec, UniXcoder, and CodeT5+, and evaluate it on a public benchmark dataset constructed from compilable Java projects. For the three-version NvRec, we find that the combination of CodeT5, MulaRec, and UniXcoder achieves the highest true accept rate of 83.8%, with a rejection rate of 80.7%, when majority voting is restricted to highly reliable candidates. In contrast, the five-version configuration achieves its highest true accept rate of 83.1% with simple majority voting, while reducing the rejection rate to 69.0%. Overall, the five-version configuration offers a better balance between true accept rate and rejection rate.