LLM-based vs. Search-based Merge Conflict Resolution: An Empirical Study of Competing Paradigms

TL;DR

This empirical study compares LLM-based and Search-Based approaches for resolving software merge conflicts, revealing their respective strengths, weaknesses, and potential for hybrid solutions.

Contribution

First comprehensive empirical comparison of LLM and SBSE paradigms in real-world merge conflict resolution scenarios.

Findings

LLM excels with imbalanced content but struggles with non-English and large inputs.

SBSE shows better generalization and performs well on balanced conflicts.

Neither approach is universally superior; hybrid methods are promising.

Abstract

Context: The resolution of software merge conflicts is being reshaped by two competing paradigms: generative approaches based on Large Language Models (LLMs) and optimization approaches from Search-Based Software Engineering (SBSE). While tools from both paradigms have shown promise, their relative strengths, weaknesses, and trade-offs are not yet well understood. Objective: This paper presents the first in-depth empirical study directly comparing these paradigms to identify their capabilities and limitations in real-world scenarios. Method: We evaluated MergeGen, a state-of-the-art LLM-based tool, against SBCR, a novel SBSE approach employing a Random Restart Hill Climbing (RRHC) algorithm. The comparison used thousands of real-world conflicts from open-source projects written in Java, C#, JavaScript, and TypeScript. Results: Our findings reveal fundamental trade-offs. The LLM paradigm excels at resolving conflicts with imbalanced content by leveraging learned patterns. However, it struggles with non-English content and large inputs, which can lead to truncated or empty resolutions. Conversely, the SBSE paradigm demonstrates superior generalization across datasets and performs best on balanced conflicts, highlighting its potential as a robust, data-independent alternative. Conclusions: Neither paradigm is a silver bullet. Our findings highlight context-dependent strengths and motivate the development of hybrid systems that combine the complementary capabilities of LLM and SBSE approaches to create more robust and reliable merge conflict resolution tools.