A Needle in a Haystack: Intent-driven Reusable Artifacts Recommendation with LLMs

TL;DR

This paper evaluates the effectiveness of Large Language Models in recommending reusable software artifacts based on developer intent, introduces a benchmark for comparison, and proposes a hierarchical framework to improve precision and efficiency.

Contribution

It constructs IntentRecBench for evaluating LLMs in artifact recommendation and proposes TreeRec, a semantic tree-guided framework that enhances LLM performance in this task.

Findings

LLMs outperform traditional methods in artifact recommendation.

TreeRec improves precision and reduces inference time.

The framework demonstrates generalizability across ecosystems.

Abstract

In open source software development, the reuse of existing artifacts has been widely adopted to avoid redundant implementation work. Reusable artifacts are considered more efficient and reliable than developing software components from scratch. However, when faced with a large number of reusable artifacts, developers often struggle to find artifacts that can meet their expected needs. To reduce this burden, retrieval-based and learning-based techniques have been proposed to automate artifact recommendations. Recently, Large Language Models (LLMs) have shown the potential to understand intentions, perform semantic alignment, and recommend usable artifacts. Nevertheless, their effectiveness has not been thoroughly explored. To fill this gap, we construct an intent-driven artifact recommendation benchmark named IntentRecBench, covering three representative open source ecosystems. Using IntentRecBench, we conduct a comprehensive comparative study of five popular LLMs and six traditional approaches in terms of precision and efficiency. Our results show that although LLMs outperform traditional methods, they still suffer from low precision and high inference cost due to the large candidate space. Inspired by the ontology-based semantic organization in software engineering, we propose TreeRec, a feature tree-guided recommendation framework to mitigate these issues. TreeRec leverages LLM-based semantic abstraction to organize artifacts into a hierarchical semantic tree, enabling intent and function alignment and reducing reasoning time. Extensive experiments demonstrate that TreeRec consistently improves the performance of diverse LLMs across ecosystems, highlighting its generalizability and potential for practical deployment.