BackportBench: A Multilingual Benchmark for Automated Backporting of Patches

TL;DR

BackportBench is a new multilingual benchmark suite designed to evaluate automated patch backporting techniques across diverse software ecosystems, highlighting the strengths and limitations of current methods.

Contribution

It introduces the first comprehensive, multilingual benchmark with executable environments for evaluating automated backporting methods on real-world problems.

Findings

Agentic methods outperform traditional patch porting techniques.

Performance varies significantly across programming languages.

BackportBench enables more realistic evaluation of backporting tools.

Abstract

Many modern software projects evolve rapidly to incorporate new features and security patches. It is important for users to update their dependencies to safer versions, but many still use older, vulnerable package versions because upgrading can be difficult and may break their existing codebase. Software developers can mitigate this problem by backporting security patches to older releases. However, manually backporting is time-consuming and error-prone. The effectiveness of existing automated backporting techniques on general software remains unclear since they typically target only code-hunk or function-level patch porting scenarios and are evaluated with imperfect metrics. To facilitate the development and evaluation of automated backporting techniques, we introduce BackportBench, the first comprehensive benchmark suite for patch backporting problem. BackportBench is a multilingual benchmark that contains 202 patch backporting problems from PyPI, Maven, and npm, each with executable Docker environments and relevant test cases. We evaluated existing patch porting methods and LLM-based techniques that have the potential to adapt to this task using BackportBench. The results show that the agentic method has outperformed traditional patch porting methods, especially on cases that require logical and structural changes. However, the performance varies across different programming languages. Based on the findings, we draw several implications for researchers and software practitioners in future work on automated backporting.