Evaluation of Version Control Merge Tools

TL;DR

This paper evaluates various version control merge tools using a realistic methodology, revealing significant differences from prior claims and introducing a new, more effective merge tool that handles common scenarios better.

Contribution

It provides a comprehensive, realistic evaluation of merge tools, and introduces a new tool that outperforms existing ones in practical scenarios.

Findings

Previous evaluations were less realistic and less comprehensive.

The new merge tool outperforms all previous tools in most scenarios.

Realistic testing methods improve the assessment of merge tool effectiveness.

Abstract

A version control system, such as Git, requires a way to integrate changes from different developers or branches. Given a merge scenario, a merge tool either outputs a clean integration of the changes, or it outputs a conflict for manual resolution. A clean integration is correct if it preserves intended program behavior, and is incorrect otherwise (e.g., if it causes a test failure). Manual resolution consumes valuable developer time, and correcting a defect introduced by an incorrect merge is even more costly. New merge tools have been proposed, but they have not yet been evaluated against one another. Prior evaluations do not properly distinguish between correct and incorrect merges, are not evaluated on a realistic set of merge scenarios, and/or do not compare to state-of-the-art tools. We have performed a more realistic evaluation. The results differ significantly from previous claims, setting the record straight and enabling better future research. Our novel experimental methodology combines running test suites, examining merges on deleted branches, and accounting for the cost of incorrect merges. Based on these evaluations, we created a merge tool that out-performs all previous tools under most assumptions. It handles the most common merge scenarios in practice.