Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training

TL;DR

This paper introduces a multi-agent reinforcement learning framework with a novel reward system for generating high-quality scientific ideas, addressing issues like reward hacking and inefficiency in existing methods.

Contribution

It presents the first multi-agent reward function as a judge for scientific ideation, decoupling validation from implementation and improving optimization against sparse rewards.

Findings

Framework outperforms baselines on novelty, feasibility, and effectiveness.

Uses an unbiased variant of Group Relative Policy Optimization.

Grounded in a curated dataset from ICLR 2024 proceedings.

Abstract

Large Language Models (LLMs) have demonstrated potential in automating scientific ideation, yet current approaches relying on iterative prompting or complex multi-agent architectures often suffer from hallucination or computational inefficiency. A critical bottleneck in applying Reinforcement Learning (RL) to this open-ended domain is reward hacking -- where models exploit imperfect evaluation proxies to maximize scores without producing genuine scientific innovation. To address these limitations, we propose an RL framework explicitly tailored for high-quality scientific idea generation. We propose the first multi-agent reward function designed to serve as a judge, decoupling methodological validation from implementation details while providing strict binary rewards that are robust to reward hacking. To effectively optimize against this sparse signal, we utilize an unbiased variant of Group Relative Policy Optimization to mitigate artificial length bias. We grounded our training in ICLR-320, a curated dataset of problem-solution pairs extracted from ICLR 2024 proceedings. Experiments demonstrate that our framework significantly outperforms state-of-the-art baselines across expert-evaluated metrics of novelty, feasibility, and effectiveness.