Studying LLM Performance on Closed- and Open-source Data

TL;DR

This study compares the performance of Large Language Models on open-source versus proprietary code, revealing that language and identifier differences impact effectiveness, with potential improvements via in-context learning.

Contribution

It provides empirical insights into LLM performance on proprietary code, highlighting language-specific effects and suggesting in-context learning as a mitigation strategy.

Findings

Performance for C# remains consistent across OSS and proprietary code.

Performance for C++ significantly decreases in proprietary code.

Differences in identifiers contribute to performance gaps.

Abstract

Large Language models (LLMs) are finding wide use in software engineering practice. These models are extremely data-hungry, and are largely trained on open-source (OSS) code distributed with permissive licenses. In terms of actual use however, a great deal of software development still occurs in the for-profit/proprietary sphere, where the code under development is not, and never has been, in the public domain; thus, many developers, do their work, and use LLMs, in settings where the models may not be as familiar with the code under development. In such settings, do LLMs work as well as they do for OSS code? If not, what are the differences? When performance differs, what are the possible causes, and are there work-arounds? In this paper, we examine this issue using proprietary, closed-source software data from Microsoft, where most proprietary code is in C# and C++. We find that performance for C# changes little from OSS --> proprietary code, but does significantly reduce for C++; we find that this difference is attributable to differences in identifiers. We also find that some performance degradation, in some cases, can be ameliorated efficiently by in-context learning.