SWE-Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering

TL;DR

SWE-Arena is an interactive, open-source platform for evaluating foundation models in software engineering, featuring novel metrics, multi-round workflows, and repository context integration to better assess model performance in real-world tasks.

Contribution

The paper introduces SWE-Arena, a novel platform with new metrics and features for comprehensive evaluation of foundation models in software engineering workflows.

Findings

Demonstrates the platform's ability to compare models effectively.

Introduces new metrics for model consistency and conversation efficiency.

Shows improved evaluation alignment with real-world SE tasks.

Abstract

Foundation models (FMs), particularly large language models (LLMs), have shown significant promise in various software engineering (SE) tasks, including code generation, debugging, and requirement refinement. Despite these advances, existing evaluation frameworks are insufficient for assessing model performance in iterative, context-rich workflows characteristic of SE activities. To address this limitation, we introduce \emph{SWE-Arena}, an interactive platform designed to evaluate FMs in SE tasks. SWE-Arena provides a transparent, open-source leaderboard, supports multi-round conversational workflows, and enables end-to-end model comparisons. The platform introduces novel metrics, including \emph{model consistency score} that measures the consistency of model outputs through self-play matches, and \emph{conversation efficiency index} that evaluates model performance while accounting for the number of interaction rounds required to reach conclusions. Moreover, SWE-Arena incorporates a new feature called \emph{RepoChat}, which automatically injects repository-related context (e.g., issues, commits, pull requests) into the conversation, further aligning evaluations with real-world development processes. This paper outlines the design and capabilities of SWE-Arena, emphasizing its potential to advance the evaluation and practical application of FMs in software engineering.