IntelliAsk: Learning to Ask High-Quality Research Questions via RLVR

TL;DR

IntelliAsk leverages reinforcement learning with a reward model trained on expert-annotated reviewer questions to generate high-quality, evidence-based research questions that outperform baselines in grounding and effort.

Contribution

The paper introduces IntelliAsk, a novel RL-based question generation model trained with a new reward model, improving the quality of research questions generated by LLMs.

Findings

IntelliAsk generates more grounded, substantive questions than baselines.

It reduces reliance on first-page content in questions.

Improves reasoning and writing benchmark scores.

Abstract

Peer review relies on substantive, evidence-based questions, yet current LLMs generate surface-level queries that perform worse than human reviewer questions in expert evaluation. To address this gap, we curate a high-quality dataset of reviewer questions from OpenReview and conduct a human preference study where expert annotators evaluate question-paper pairs across three dimensions: effort, evidence, and grounding. From these annotations, we train IntelliReward, a reward model built from a frozen autoregressive LLM with trainable multi-head transformers. Validated against expert judgments, IntelliReward predicts reviewer-question quality better than API-based SFT baselines and provides scalable evaluation. We apply Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) with IntelliReward to train IntelliAsk, a question-generation model aligned with human standards of effortful, evidence-based critique. Human evaluations show IntelliAsk generates more grounded, substantive and effortful questions than strong baselines and reduces reliance on first-page content. We also find improvements on reasoning and writing benchmarks, suggesting reviewer-question quality correlates with broader capabilities. Compared to Qwen3-32B, IntelliAsk improves MuSR (68.3 vs 64.7 Acc) and WritingBench (8.31 vs 8.07). We release our code, filtered review dataset, expert annotations, IntelliAsk and IntelliReward to support automatic evaluation of grounding, effort, and evidence in LLM-generated review questions.