Generating Unseen Code Tests In Infinitum
Marcel Zalmanovici, Orna Raz, Eitan Farchi, Iftach Freund

TL;DR
This paper introduces a method for generating diverse, unseen code tests to evaluate large language models' coding abilities, addressing data leakage issues in benchmarks and supporting ongoing model assessment.
Contribution
The authors propose a novel approach for creating adaptable, non-leaking benchmark variations applicable across coding tasks and languages, including in-house code bases.
Findings
Developed 'auto-regression' benchmark for Python code generation.
Enables continuous testing and debugging of LLMs.
Mitigates data leakage in model evaluation.
Abstract
Large Language Models (LLMs) are used for many tasks, including those related to coding. An important aspect of being able to utilize LLMs is the ability to assess their fitness for specific usages. The common practice is to evaluate LLMs against a set of benchmarks. While benchmarks provide a sound foundation for evaluation and comparison of alternatives, they suffer from the well-known weakness of leaking into the training data \cite{Xu2024Benchmarking}. We present a method for creating benchmark variations that generalize across coding tasks and programming languages, and may also be applied to in-house code bases. Our approach enables ongoing generation of test-data thus mitigating the leaking into the training data issue. We implement one benchmark, called \textit{auto-regression}, for the task of text-to-code generation in Python. Auto-regression is specifically created to aid in…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEducational Technology and Assessment
MethodsSparse Evolutionary Training
