LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations

TL;DR

LibRec is a framework that combines retrieval-augmented generation with in-context learning to improve library migration recommendations, evaluated on a new benchmark called LibEval with extensive experiments.

Contribution

The paper introduces LibRec, a novel framework integrating LLMs and retrieval techniques for library migration, along with LibEval, a comprehensive benchmark for evaluation.

Findings

LibRec outperforms baseline models in migration accuracy.

Prompt strategies significantly influence recommendation performance.

The framework effectively handles various migration intent types.

Abstract

In this paper, we propose LibRec, a novel framework that integrates the capabilities of LLMs with retrieval-augmented generation(RAG) techniques to automate the recommendation of alternative libraries. The framework further employs in-context learning to extract migration intents from commit messages to enhance the accuracy of its recommendations. To evaluate the effectiveness of LibRec, we introduce LibEval, a benchmark designed to assess the performance in the library migration recommendation task. LibEval comprises 2,888 migration records associated with 2,368 libraries extracted from 2,324 Python repositories. Each migration record captures source-target library pairs, along with their corresponding migration intents and intent types. Based on LibEval, we evaluated the effectiveness of ten popular LLMs within our framework, conducted an ablation study to examine the contributions of key components within our framework, explored the impact of various prompt strategies on the framework's performance, assessed its effectiveness across various intent types, and performed detailed failure case analyses.