Using Copilot Agent Mode to Automate Library Migration: A Quantitative Assessment

TL;DR

This paper evaluates the use of GitHub Copilot's Agent Mode to automate library migration in Python applications, demonstrating high API migration coverage but limited success in preserving application functionality.

Contribution

It introduces a novel application of LLM-based autonomous agents for library migration and proposes the Migration Coverage metric for assessment.

Findings

API migration coverage median of 100%

Test-pass rate median of 39.75%

Automated migration can be highly accurate but may not preserve functionality

Abstract

Keeping software systems up to date is essential to avoid technical debt, security vulnerabilities, and the rigidity typical of legacy systems. However, updating libraries and frameworks remains a time consuming and error-prone process. Recent advances in Large Language Models (LLMs) and agentic coding systems offer new opportunities for automating such maintenance tasks. In this paper, we evaluate the update of a well-known Python library, SQLAlchemy, across a dataset of ten client applications. For this task, we use the Github's Copilot Agent Mode, an autonomous AI systema capable of planning and executing multi-step migration workflows. To assess the effectiveness of the automated migration, we also introduce Migration Coverage, a metric that quantifies the proportion of API usage points correctly migrated. The results of our study show that the LLM agent was capable of migrating functionalities and API usages between SQLAlchemy versions (migration coverage: 100%, median), but failed to maintain the application functionality, leading to a low test-pass rate (39.75%, median).