TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks

TL;DR

TimeMachine-bench is a new benchmark for assessing the ability of models to perform software migration tasks in real-world Python projects, focusing on dependency update failures.

Contribution

It introduces an automated, live-updating benchmark with a human-verified subset for evaluating model performance on repository-level migration tasks.

Findings

LLMs show some promise but face reliability issues.

Models often produce spurious solutions exploiting low test coverage.

Suboptimal tool-use strategies lead to unnecessary edits.

Abstract

With the advancement of automated software engineering, research focus is increasingly shifting toward practical tasks reflecting the day-to-day work of software engineers. Among these tasks, software migration, a critical process of adapting code to evolving environments, has been largely overlooked. In this study, we introduce TimeMachine-bench, a benchmark designed to evaluate software migration in real-world Python projects. Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates. The construction process is fully automated, enabling live updates of the benchmark. Furthermore, we curated a human-verified subset to ensure problem solvability. We evaluated agent-based baselines built on top of 11 models, including both strong open-weight and state-of-the-art LLMs on this verified subset. Our results indicated that, while LLMs show some promise for migration tasks, they continue to face substantial reliability challenges, including spurious solutions that exploit low test coverage and unnecessary edits stemming from suboptimal tool-use strategies. Our dataset and implementation are available at https://github.com/tohoku-nlp/timemachine-bench.