Pareto Optimal Code Generation

TL;DR

This paper introduces a Pareto optimization framework for code generation verification, demonstrating that staged verification with outcome reward models significantly improves throughput with minimal accuracy loss.

Contribution

It formulates verifier selection as a Pareto optimization problem and empirically maps the accuracy-throughput frontier for various signals, highlighting the effectiveness of staged verification with ORMs.

Findings

ORMs are most effective with staged verification.

Staged verification achieves 11.64x higher throughput.

Minimal 8.26% accuracy loss with ORM-based staged verification.

Abstract

Generate-then-rank is the dominant test-time scaling (TTS) paradigm for code generation, but scaling accuracy by sampling and executing more candidates makes comprehensive verification a major computational bottleneck. This creates an inherent trade-off between accuracy and compute that, despite its importance to TTS, is often ignored. Specifically, faster but noisier signals, such as outcome reward models (ORMs), are dismissed as suboptimal. We frame verifier selection as a Pareto optimization problem and empirically map the accuracy-throughput frontier across signals, including the full test suite, heuristics for selective execution, and ORMs, across four Python benchmarks. We show that ORMs are most effective at optimizing the Pareto curve when pruning is used in the generate-then-rank pipeline--known as staged verification--where lightweight filters remove obviously incorrect solutions, including candidates with small syntactic or character-level bugs, before expensive verification. Our pruning analysis shows that eliminating incorrect yet highly ranked candidates (often character-level bugs) prevents wasted compute on incorrect tokens. We find that ORMs with staged verification shift the Pareto frontier outward, achieving 11.64x higher throughput at a cost of 8.26% accuracy relative to full test-suite verification.