SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks

TL;DR

SWE-Sharp-Bench introduces a comprehensive, reproducible benchmark for evaluating AI coding agents on C# tasks, highlighting a performance gap compared to Python and promoting open science.

Contribution

It provides the first extensive C# software engineering benchmark with 150 instances, enabling cross-language evaluation and reproducibility.

Findings

70% of Python tasks solved in SWE-Bench Verified

Only 40% of C# tasks solved, indicating a performance gap

Open-sourced benchmark and curation pipeline

Abstract

AI coding agents have shown great progress on Python software engineering benchmarks like SWE-Bench, and for other languages like Java and C in benchmarks like Multi-SWE-Bench. However, C# -- a prominent enterprise language ranking #5 in the TIOBE index -- remains absent from such benchmarks. We introduce SWE-Sharp-Bench, a reproducible software engineering benchmark for C# featuring 150 instances from 17 repositories. Evaluating identical model-agent configurations across languages reveals a significant performance gap: while 70% of Python tasks in SWE-Bench Verified are solved, only 40% of our C# tasks are resolved. We open-source SWE-Sharp-Bench and our entire curation pipeline.