RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks

TL;DR

RLAC introduces a dynamic adversarial critic framework using large language models to improve free-form generation tasks by efficiently identifying failure modes and enhancing output quality, reducing verification costs.

Contribution

This paper presents RLAC, a novel reinforcement learning approach with a dynamic adversarial critic that improves free-form generation by focusing on likely failure modes, enabling scalable and effective post-training optimization.

Findings

RLAC improves factual accuracy in text generation.

RLAC enhances correctness in code generation.

Dynamic critics outperform fixed critics in verification efficiency.

Abstract

Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to prohibitively high verification costs and incomplete assessments of a response, making reinforcement learning (RL) post-training with rubric-based rewards difficult to scale. This problem is exacerbated by the fact that often the best way to combine these rubrics into one single reward is also highly prompt-specific. We propose Reinforcement Learning with Adversarial Critic (RLAC), a post-training approach that addresses these challenges via dynamic rubric verification. Our approach employs a large language model (LLM) as a critic that dynamically identifies only the most likely failure modes (e.g., a factual error or unhandled edge case), which are then verified by an external validator to optimize both generator and critic jointly. By training both the generator and the critic, this game enhances the critic's error detection and the generator's output quality while reducing required verifications. Our experiments demonstrate that RLAC improves factual accuracy in text generation and correctness in code generation, while also outperforming exhaustive verification and reward model methods. We show that dynamic critics are more effective than fixed critics, showcasing the potential of RLAC for scaling RL post-training to free-form generation tasks.