OSS-Bench: Benchmark Generator for Coding LLMs

TL;DR

OSS-Bench is a scalable benchmark generator that evaluates LLMs on real-world open-source software tasks, focusing on code correctness, compilability, and memory safety, revealing insights into model behavior and security understanding.

Contribution

It introduces OSS-Bench, a novel automated framework for large-scale, live evaluation of coding LLMs using real OSS code and multiple robust metrics.

Findings

Profiles 17 diverse LLMs revealing behavioral patterns

Identifies inconsistencies between model size and performance

Highlights LLMs' limited understanding of low-level security

Abstract

In light of the rapid adoption of AI coding assistants, LLM-assisted development has become increasingly prevalent, creating an urgent need for robust evaluation of generated code quality. Existing benchmarks often require extensive manual effort to create static datasets, rely on indirect or insufficiently challenging tasks, depend on non-scalable ground truth, or neglect critical low-level security evaluations, particularly memory-safety issues. In this work, we introduce OSS-Bench, a benchmark generator that automatically constructs large-scale, live evaluation tasks from real-world open-source software. OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth. In our evaluation, the benchmark, instantiated as OSS-Bench(php) and OSS-Bench(sql), profiles 17 diverse LLMs, revealing insights such as intra-family behavioral patterns and inconsistencies between model size and performance. Our results demonstrate that OSS-Bench mitigates overfitting by leveraging the evolving complexity of OSS and highlights LLMs' limited understanding of low-level code security via extended fuzzing experiments. Overall, OSS-Bench offers a practical and scalable framework for benchmarking the real-world coding capabilities of LLMs.