Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation

TL;DR

This paper introduces Stream of Revision, a novel approach enabling large language models to perform self-correcting code generation within a single pass, improving security and reducing vulnerabilities.

Contribution

It presents a new paradigm and action tokens that allow models to backtrack and revise code during generation, internalizing the revision process without external tools.

Findings

Significantly reduces vulnerabilities in secure code generation.

Achieves this with minimal inference overhead.

Demonstrates the effectiveness of internal revision in LLMs.

Abstract

Large Language Model (LLM) based code generation is predominantly formulated as a strictly monotonic process, appending tokens linearly to an immutable prefix. This formulation contrasts to the cognitive process of programming, which is inherently interleaved with forward generation and on-the-fly revision. While prior works attempt to introduce revision via post-hoc agents or external static tools, they either suffer from high latency or fail to leverage the model's intrinsic semantic reasoning. In this paper, we propose Stream of Revision, a paradigm shift that elevates code generation from a monotonic stream to a dynamic, self-correcting trajectory by leveraging model's intrinsic capabilities. We introduce specific action tokens that enable the model to seamlessly backtrack and edit its own history within a single forward pass. By internalizing the revision loop, our framework Stream of Revision allows the model to activate its latent capabilities just-in-time without external dependencies. Empirical results on secure code generation show that Stream of Revision significantly reduces vulnerabilities with minimal inference overhead.