RubberDuckBench: A Benchmark for AI Coding Assistants

TL;DR

RubberDuckBench is a multilingual benchmark of real-world questions from GitHub, designed to evaluate AI coding assistants, revealing current models' limitations in accuracy, consistency, and hallucination rates.

Contribution

This work introduces RubberDuckBench, a new benchmark with detailed rubrics for evaluating AI coding assistants on real-world, multilingual code-related questions.

Findings

State-of-the-art models perform poorly on the benchmark.

Most models only answer a few questions completely correctly.

Models frequently hallucinate, with lies in 58.3% of responses.

Abstract

Programmers are turning to AI coding assistants to answer questions about their code. Benchmarks are needed to soundly evaluate these systems and understand their performance. To enable such a study, we curate a benchmark of real-world contextualized questions derived from Github pull request comments. Out of this work, we present RubberDuckBench: a multilingual benchmark of questions about code, along with detailed rubrics for evaluating answers. We evaluate a diverse set of 20 LLMs (proprietary & open-source) on answering these questions. We find that even state of the art models fail to give consistent, correct responses across the benchmark. Grok 4 (69.29%), Claude Opus 4 (68.5%), and GPT-5 (67.8%) perform best overall, but do not exhibit pairwise significant superiority over the next 9 best performing models. Most models obtain points through partial credit, with the best performing models only answering at most 2 questions completely correctly across all trials. Furthermore, models often hallucinate with lies in 58.3\% of responses on average. Cost analysis reveals no correlation between expense (API pricing or parameter count) and performance. We intend this benchmark to be a target for future research in trustworthy and correct AI coding assistants.