Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot

TL;DR

This study empirically evaluates how often links provided by Bing CoPilot and Google Gemini are relevant to generated code snippets, revealing significant issues with provenance and trustworthiness in LLM-based code assistants.

Contribution

It provides the first systematic analysis of link relevance in LLM-generated code, highlighting the extent of provenance debt and its implications.

Findings

66% of Bing CoPilot links are relevant

28% of Google Gemini links are relevant

LLMs often provide irrelevant or misleading links

Abstract

Large Language Models (LLMs) are currently used for various software development tasks, including generating code snippets to solve specific problems. Unlike reuse from the Web, LLMs are limited in providing provenance information about the generated code, which may have important trustworthiness and legal consequences. While LLM-based assistants may provide external links that are "related" to the generated code, we do not know how relevant such links are. This paper presents the findings of an empirical study assessing the extent to which 243 and 194 code snippets, across six programming languages, generated by Bing CoPilot and Google Gemini, likely originate from the links provided by these two LLM-based assistants. The study leverages automated code similarity assessments with thorough manual analysis. The study's findings indicate that the LLM-based assistants provide a mix of relevant and irrelevant links having a different nature. Specifically, although 66% of the links from Bing CoPilot and 28% from Google Gemini are relevant, LLMs-based assistants still suffer from serious "provenance debt".