Addressing Popularity Bias in Third-Party Library Recommendations Using LLMs

TL;DR

This paper investigates whether large language models can mitigate popularity bias in third-party library recommendations, finding that current LLMs are ineffective despite some improvements in recommendation diversity.

Contribution

The study evaluates state-of-the-art LLM techniques for reducing popularity bias in software library recommenders, highlighting their limitations and suggesting directions for future research.

Findings

LLMs do not effectively address popularity bias in TPL recommenders.

Fine-tuning and penalty mechanisms increase recommendation diversity.

Current LLMs have limitations in mitigating popularity bias.

Abstract

Recommender systems for software engineering (RSSE) play a crucial role in automating development tasks by providing relevant suggestions according to the developer's context. However, they suffer from the so-called popularity bias, i.e., the phenomenon of recommending popular items that might be irrelevant to the current task. In particular, the long-tail effect can hamper the system's performance in terms of accuracy, thus leading to false positives in the provided recommendations. Foundation models are the most advanced generative AI-based models that achieve relevant results in several SE tasks. This paper aims to investigate the capability of large language models (LLMs) to address the popularity bias in recommender systems of third-party libraries (TPLs). We conduct an ablation study experimenting with state-of-the-art techniques to mitigate the popularity bias, including fine-tuning and popularity penalty mechanisms. Our findings reveal that the considered LLMs cannot address the popularity bias in TPL recommenders, even though fine-tuning and post-processing penalty mechanism contributes to increasing the overall diversity of the provided recommendations. In addition, we discuss the limitations of LLMs in this context and suggest potential improvements to address the popularity bias in TPL recommenders, thus paving the way for additional experiments in this direction.