Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems

TL;DR

This paper evaluates the ability of large language models to generate reliable test case generators for competitive programming problems, introducing a benchmark and analyzing their strengths and limitations in generating both valid and targeted test cases.

Contribution

It introduces TCGBench, a new benchmark for testing LLMs in generating test case generators for competitive programming, and provides insights into their capabilities and challenges.

Findings

LLMs can generate valid test case generators in most cases.

LLMs struggle to generate targeted test cases that expose bugs.

Performance can be improved with curated instruction datasets.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, capable of tackling complex tasks during inference. However, the extent to which LLMs can be utilized for code checking or debugging through test case generation remains largely unexplored. We investigate this problem from the perspective of competition-level programming (CP) programs and propose TCGBench, a Benchmark for (LLM generation of) Test Case Generators. This benchmark comprises two tasks, aimed at studying the capabilities of LLMs in (1) generating valid test case generators for a given CP problem, and further (2) generating targeted test case generators that expose bugs in human-written code. Experimental results indicate that while state-of-the-art LLMs can generate valid test case generators in most cases, most LLMs struggle to generate targeted test cases that reveal flaws in human code effectively. Especially, even advanced reasoning models (e.g., o3-mini) fall significantly short of human performance in the task of generating targeted generators. Furthermore, we construct a high-quality, manually curated dataset of instructions for generating targeted generators. Analysis demonstrates that the performance of LLMs can be enhanced with the aid of this dataset, by both prompting and fine-tuning.