Hydra: Efficient, Correct Code Generation via Checkpoint-and-Rollback Support

TL;DR

Hydra is a system that improves code generation efficiency by enabling asynchronous error checking and targeted repair, significantly reducing latency and token use in C/C++ code generation.

Contribution

Hydra introduces checkpoint-and-rollback support and asynchronous checking to enhance static error recovery during code generation, reducing overhead and improving efficiency.

Findings

Reduces latency by up to 71% in code generation tasks.

Lowers token consumption by up to 70% with Hydra.

Supports efficient static error recovery in C/C++ code generation.

Abstract

Large language models are increasingly used for code generation, but many generated programs fail to compile, a prerequisite for further correctness checks such as unit tests. Existing solutions for repairing static errors are costly in both latency and token consumption. Post-hoc repair delays error detection until generation completes and commonly regenerates large regions of previously valid code. Constrained semantic decoding checks after each token, incurring per-token overhead while limiting repair to the current token even when the root cause lies earlier. We present Hydra, a system for efficient recovery from static errors during code generation. Hydra allows checking to proceed asynchronously with generation, avoiding checker overhead when the generated code is semantically correct. In addition, it provides checkpoint-and-rollback support for targeted repair, avoiding regeneration and rechecking of valid prefixes. We retrofit the Clang C/C++ compiler to support Hydra with modest modifications. Paired with a token-efficient repair strategy, Hydra reduces latency by up to 71% and token consumption by up to 70% relative to post-hoc repair on C/C++ code generation tasks that encounter static errors.