ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics

Li S. Yifei; Allen Chang; Chaitanya Malaviya; Mark Yatskar

arXiv:2509.00496·cs.CL·December 22, 2025

ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics

Li S. Yifei, Allen Chang, Chaitanya Malaviya, Mark Yatskar

PDF

Open Access 1 Datasets

TL;DR

ResearchQA introduces a large-scale dataset derived from survey articles across 75 fields to evaluate scholarly question answering systems, highlighting current system limitations in coverage and answer quality.

Contribution

The paper presents ResearchQA, a novel resource for evaluating research-focused question answering systems using survey-mined questions and rubrics across multiple disciplines.

Findings

01

No system exceeds 70% coverage of rubric items

02

The top system fully addresses less than 11% of citation rubrics

03

Error analysis identifies key areas for improvement in system responses

Abstract

Evaluating long-form responses to research queries heavily relies on expert annotators, restricting attention to areas like AI where researchers can conveniently enlist colleagues. Yet, research expertise is abundant: survey articles consolidate knowledge spread across the literature. We introduce ResearchQA, a resource for evaluating LLM systems by distilling survey articles from 75 research fields into 21K queries and 160K rubric items. Queries and rubrics are jointly derived from survey sections, where rubric items list query-specific answer evaluation criteria, i.e., citing papers, making explanations, and describing limitations. 31 Ph.D. annotators in 8 fields judge that 90% of queries reflect Ph.D. information needs and 87% of rubric items warrant emphasis of a sentence or longer. We leverage ResearchQA to evaluate 18 systems in 7.6K head-to-heads. No parametric or…

Tables13

Table 1. Table 1: Comparison of ResearchQA to related benchmarks. Icons: Health Sciences & Medicine; Life & Earth Sciences; Engineering & CS; Physical Sciences; Social Sciences; Humanities; Economics. (1) qasper ; (2) qasa ; (3) pubmedqa ; (4) sciqa ; (5) kiwi ; (6) scidqa ; (7) sciqag ; (8) openscholar ; (9) sciarena ( † Concurrent work). Scholarly fields are marked as blank “—” when field descriptions are incomplete or metadata are missing.

Table 2. Table 2: Inter-annotator agreement across experts.

Section	Annotation Type	PA
Section˜3.1	Clarity	.72
	Naturalness	.73
	Relevance	.83
	Variability in Valid Answers	.88
Section˜3.2	Proportion of Answer	.91
Section˜3.2	Omission Based Change in Answer Quality	.94
Section˜3.3	Rubric Coverage	.73
Section˜3.3	Pairwise Preference	.69

Table 3. Table 3: Performances of LLM systems in a pairwise tournament across domains ( Health Sciences & Medicine; Life & Earth Sciences; Engineering & CS; Physical Sciences; Social Sciences; Humanities; Economics.) Retrieval embedding models belong to corresponding system providers: i openscholar-retrieve and openscholar-reranker ; ii text-embedding-004 ; iii gte-qwen-2-7b-instruct ; iv voyage-3-large ; v text-embedding-3-large . † This system costs ∼ \sim $1.15 per query, so statistics are only computed on answers sampled for tournament battles ( ∼ \sim 20%).

Table 4. Table 4: Removing the L L =250 words length guidance prompt can affect average answer length (words). Longer answers tend to score higher coverage, because coverage is a recall-based measure. Analysis is performed on a 225 query subset of 𝒟 test \mathcal{D}_{\text{test}} (3 per field).

Systems	Avg Length			Coverage %
	$L$ =250	$\neg L$	$Δ$ %	$L$ =250	$\neg L$	$Δ$
Parametric
qwen-3-32b	219.5	0176.5	0 $-$ 19%	66.8	64.0	$-$ 2.8
gemini-2.5-pro	268.5	0241.4	0 $-$ 10%	69.9	67.5	$-$ 2.5
Retrieval (Naive)
claude-4-sonnet	238.9	0299.5	0 $+$ 25%	63.6	62.8	$-$ 0.8
gpt-4.1	264.0	0246.1	00 $-$ 6%	65.0	63.7	$-$ 1.3
Deep Research
o4-mini-deep-research	268.0	0595.0	$+$ 122%	72.2	78.9	$+$ 6.7
sonar-deep-research	267.2	1431.0	$+$ 435%	76.2	85.3	$+$ 9.1

Table 5. Table 5: Error rates for different rubric types, measured as the percentage of items not rated as Completely covered by the best-performing system ( sonar-deep-research ). Each rubric type provides a description (with X X representing a concept, method, or paper being evaluated) along with an illustrated example.

Item Type	Description	Example	Frequency %	Error %
Citation	$X$ is cited	Does the response cite Kvaskoff et al. (2015) (title: […]) that links endometriosis with elevated cardiovascular risk?	8.3	89.3
Limitation	Limitations of $X$ are mentioned	Does the response address limitations of CTC in detecting small polyps and flat adenomas?	2.7	52.4
Comparison	$X$ and $Y$ are compared	Does the response compare reaction rates before and after catalyst saturation occurs?	14.2	51.9
Example	Examples of $X$ are mentioned	Does the response include forage species (e.g., legumes, chicory) affecting lamb meat’s fatty acid profile?	11.2	46.8
Impact	Cause or impact of $X$ is mentioned	Does the response mention the preservation of the anterior cruciate ligament (ACL) as a benefit of UKA?	15.5	46.3
Other	None of the above	Does the response discuss METEOR’s fragmentation penalty and its role in evaluating word order?	48.1	43.6

Table 6. Table 6: Production systems often cite the related survey as a source, at 20-30% leakage (L%). However, Coverage % roughly stays the same with leakage (Leaked) and without leakage ( ¬ \neg Leaked).

System	L%	Coverage %
		$\neg$ Leaked	Leaked	$Δ$
claude-4-sonnet+ws	30	71.27	68.30	$-$ 3.0
gemini-2.5-pro+grounding	2	66.91	68.54	$+$ 1.6
o4-mini-deep-research	28	74.82	71.85	$-$ 3.0
sonar-deep-research	21	74.46	75.51	$+$ 1.1

Table 7. Table 7: Aggregated numbers for top-20 publishing venues in Google Scholar categories.

Field	Fields	Venues
Business, Economics, & Management	16	285
Chemical & Material Sciences	17	318
Physics & Mathematics	21	370
Humanities, Literature, & Arts	26	481
Life Sciences	30	509
Social Sciences	31	526
Engineering & Computer Science	50	925
Health & Medical Sciences	66	1220
Total	257	4634

Table 8. Table 8: The number of queries for each of 75 fields, by research domain, in ResearchQA .

Table 9. Table 9: Comparisons between when pipeline models are used as generators. Generally, queries and rubrics are comparable in quality, where those from gpt-4.1 are slightly more specific than those from gpt-4.1-mini .

Type	gpt-4.1-mini	gpt-4.1
$Q$	What is the rule-based approach in machine translation, and what are its benefits and limitations?	How do rule-based machine translation systems use linguistic analysis and dictionaries to carry out translations?
$R_{1}$	Does the response provide a detailed explanation of the linguistic analysis process involved in the Rule-based Approach, including morphology, syntax, and lexical semantics?	Does the response explain how linguistic analysis at multiple levels (morphology, syntax, lexical semantics) is used in rule-based machine translation?
$R_{2}$	Does the response discuss the flexibility of the Rule-based Approach in adapting to new language constructs by updating the dictionary?	Does the response describe the use of bilingual dictionaries for mapping source language words to target language equivalents?
$R_{3}$	Does the response highlight the advantage of the Rule-based Approach in not requiring as many parallel sentence pairs as Neural Machine Translation?	Does the response discuss how RBMT systems handle word ambiguities and parts of speech during translation?
$R_{4}$	Does the response mention the use of dictionaries and expert knowledge in establishing grammar rules for the Rule-based Approach?	Does the response explore in depth how linguistic analysis at multiple levels (morphology, syntax, lexical semantics) is performed and integrated in rule-based machine translation systems?
$R_{5}$	Does the response include a definition or explanation of what a Rule-based Approach in machine translation is?	Does the response provide a detailed explanation of how bilingual dictionaries are structured and utilized within the translation process?
$R_{6}$	Does the response mention specific benefits of using a Rule-based Approach in machine translation?	Does the response provide a detailed explanation of how linguistic analysis components are integrated into the translation process?
$R_{7}$	Does the response highlight any limitations or challenges associated with the Rule-based Approach in machine translation?	Does the response explore in depth the role and structure of dictionaries used in rule-based machine translation?
$R_{8}$	Does the response cite the paper by Scott and Barreiro (2009) (title: OpenLogos MT and the SAL representation language) that provides insights into the linguistic analysis and dictionary mapping process in Rule-based Machine Translation?	Does the response cite papers like Kay (1980) (title: The Proper Place of Men and Machines in Language Translation) that discuss the integration of dictionaries with linguistic rules for accurate translation?

Table 10. Table 10: Cost comparison for different pipeline models for the full generation of ResearchQA . All numbers are in batched inference pricing (2 × \times cheaper).

Model	Queries	Rubrics	Total
gpt-4.1-mini	$ 0349.65	$ 133.65	$ 0483.30
gpt-4.1	$ 1748.25	$ 668.25	$ 2416.50
gpt-4o	$ 2185.31	$ 835.31	$ 3020.62

Table 11. Table 11: The generic rubric baseline, sampled from ScholarQABench openscholar and SciArena sciarena criteria. “Examples” is manually written to fill the rubric to 8 items.

Type	Generic Rubric Item
Correctness	Does the response make factually correct statements?
Citations	Does the response cite relevant papers?
Coverage	Does the response provide sufficient coverage and amount of information?
Relevance	Does the response stay on topic and maintain a clear focus?
Organization	Does the response have an organized structure?
Usefulness	Does the response fulfill information needs?
Attribution	Does the response make correct citation-to-claim attributions?
Examples	Does the response include relevant examples?

Table 12. Table 12: LLM System Configuration and Cost. Default setting: temperature=0 . Cost is for running on the full test set of ResearchQA (3750 queries). Organizations and related works: Meta grattafiori2024llama , Anthropic claude , OpenAI gpt4 ; openaiIntroducingDeep , Alibaba yang2025qwen3 , Google geminitwofive ; geminiGeminiDeep , AI2 openscholar , Perplexity perplexity2025deepresearch .

Table 13. Table 13: Retrieving the distilled survey does not provide large advantages toward higher Coverage %. Parametric systems have 3-8% leakage (L%); Retrieval (Naive) do not produce answers with leakage (likely because surveys are intentionally removed from their retrieval); Retrieval (Production) and Deep Research result in 20-30% leakage. Coverage % roughly stays the same with leakage (Leaked) and without leakage ( ¬ \neg Leaked).

Equations2

4 (I_{A} + I_{A}^{'}) + \sum_{c \in Coverage (A, R)} c

4 (I_{A} + I_{A}^{'}) + \sum_{c \in Coverage (A, R)} c

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Code & Models

Datasets

realliyifei/ResearchQA
dataset· 294 dl
294 dl

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Taxonomy

TopicsExpert finding and Q&A systems · Topic Modeling · Information Retrieval and Search Behavior

Full text

ResearchQA: Evaluating Scholarly Question Answering at Scale

Across 75 Fields with Survey-Mined Questions and Rubrics

Li S. Yifei∗, Allen Chang∗, Chaitanya Malaviya, Mark Yatskar

University of Pennsylvania

{liyifei, cylumn}@seas.upenn.edu

Data: huggingface.co/datasets/realliyifei/ResearchQA

Code: github.com/realliyifei/ResearchQA

Website: cylumn.com/ResearchQA

Abstract

Evaluating long-form responses to research queries heavily relies on expert annotators, restricting attention to areas like AI where researchers can conveniently enlist colleagues. Yet, research expertise is abundant: survey articles consolidate knowledge spread across the literature. We introduce ResearchQA, a resource for evaluating LLM systems by distilling survey articles from 75 research fields into 21K queries and 160K rubric items. Queries and rubrics are jointly derived from survey sections, where rubric items list query-specific answer evaluation criteria, i.e., citing papers, making explanations, and describing limitations. 31 Ph.D. annotators in 8 fields judge that 90% of queries reflect Ph.D. information needs and 87% of rubric items warrant emphasis of a sentence or longer. We leverage ResearchQA to evaluate 18 systems in 7.6K head-to-heads. No parametric or retrieval-augmented system we evaluate exceeds 70% on covering rubric items, and the highest-ranking system shows 75% coverage. Error analysis reveals that the highest-ranking system fully addresses less than 11% of citation rubric items, 48% of limitation items, and 49% of comparison items. We release our data to facilitate more comprehensive multi-field evaluations.

{NoHyper}††footnotetext: ∗Equal contribution.

1 Introduction

The rapid growth in research literature makes staying informed about advancements in many fields difficult price1963little; larsen2010rate. Large language model (LLM) tools, such as deep research systems geminiGeminiDeep; openaiIntroducingDeep and scientific AI assistants skarlinski2024language; sciencediscovery; si2024llmideas; singh2025ai2, show potential to address this problem by meeting the information needs of both experts and non-experts. However, evaluating long form answers to research queries is extremely challenging kiwi. Several benchmarks have been proposed (i.a., qasa; sciqa; openscholar; sciarena), but are limited in size and primarily constrained to engineering domains (Table˜1). Broader evaluation is necessary, but, as yet, has been unachievable because of a lack of affordable availability of appropriate experts.

In this paper, we introduce ResearchQA, a literature-based resource for benchmarking research synthesis systems. Our insight is to leverage academic surveys, whose role is to review a research field’s foundational questions and synthesize relevant evidence kasanishi2023scireviewgen, for conveniently making comprehensive evaluation of research topics possible. We operationalize this idea by building a multi-stage pipeline to transform a pool of over 54K academic surveys into queries and answer evaluation rubrics lin2024wildbench; sawadaarb. Each rubric, derived jointly with queries by mining high quality survey sections, lists query-specific evaluation criteria, which may include citing papers, and making comparisons, or describing causal effects, among others (Figure˜1). To demonstrate the effectiveness of literature-based evaluation, we release ResearchQA, containing 21.4K queries with 160k rubric items, across 75 research fields, making it the most diverse and comprehensive benchmark of its kind to date (Tables˜1 and 8).

We validate the quality of ResearchQA in a multi-field expert evaluation, spanning 31 Ph.D. level annotators with expertise from 8 fields. 90% of queries are judged as reflecting information needs of Ph.D. students and 86% are stylistically similar to how a Ph.D. student would have expressed that need. Few queries are considered too open-ended, facilitating the opportunity for criteria-based evaluation over open interpretation. We also evaluate the quality of rubric items, finding that 87% of them address concepts that should be covered in at least a sentence by systems.

We explore whether rubrics can improve LLM-as-a-Judge accuracy. We sample research queries and ask the same annotators to vote for the higher response quality between two retrieval-augmented systems. Expert preferences are aggregated using majority voting and find that human accuracy to these labels is 84%, which approximates an upper bound for LLM-as-a-Judge accuracy. Proprietary and open-sourced judges that directly predict preference show 71% and 64% accuracy respectively, likely because they lack appropriate expert knowledge. To bridge this knowledge gap, we score each response from 0 to 4 for each rubric item’s coverage (Figure˜1). To our excitement, we find that agreement increases to 74% and 69% when coverage and direct predictions are combined in an ensemble LLM-as-a-Judge protocol.

With ResearchQA, we extensively evaluate frontier systems, revealing significant performance gaps between how different inference pipelines address scholarly queries. We consider 18 systems across four tiers of test-time inference effort: language models using solely parametric memory, naive retrieval on Google Scholar, production-level retrieval optimized for consumer use, and deep research APIs. Each system is judged on two expert-validated evaluation metrics: (1) average rubric coverage %, and (2) Elo score using our ensemble LLM-as-a-Judge. The results reveal four noteworthy observations:

All systems struggle to cover rubric items. No parametric or retrieval system we evaluate exceeds 70% coverage. The highest-performing system we evaluate is Perplexity’s deep research perplexity2025deepresearch, with 75.29% coverage, indicating room for improvement.

Models selectively benefit from retrieval. When only relying on parametric memory, Claude-Sonnet-4 claude ranks lower than Gemini-2.5-Pro geminitwofive (30% win rate), but when both models use retrieval Claude-Sonnet-4 ranks higher (75% win rate).

Retrieval optimization is critical. We observe that a naive retrieval implementation is often insufficient to improve coverage. On average, the same models differ by 7.6% coverage between naive and production retrieval.

Deep research systems are significantly advantaged for scholarly queries. Strikingly, Perplexity’s deep research obtains 82% win rate over the next best system, revealing substantial gaps between the helpfulness of research-oriented tools.

Finally, we conduct an error breakdown of unaddressed rubric items, revealing that systems can improve across many criteria: common error cases include citing key works (under-addressed in 89% of cases), describing limitations (52% of cases), and making comparisons (51% of cases).

2 ResearchQA: Mining Queries and Rubrics from Academic Surveys

ResearchQA is a large-scale research question answering dataset, consisting of 21.4K queries across 75 research fields. Research fields span 7 domains: Health Sciences & Medicine (7.5K queries), Life & Earth Sciences (4.9K), Engineering & Computer Science (4.7K), Physical Sciences (2.5K), Social Sciences (1.4K), Humanities (362), and Economics (55). Table˜1 shows that ResearchQA contains similar desiderata of related benchmarks, while expanding on queries and research field diversity. Table˜8 shows the number of queries for each field.

To build ResearchQA, we create a multi-stage pipeline to generate queries and rubrics from survey articles (Figure˜2). We describe the selection of top publication venues in Section˜2.1 and survey articles in Section˜2.2. We present query generation in Section˜2.3 and rubric generation in Section˜2.4. Finally, we describe dataset splits in Section˜2.5. Further details about pipeline implementation are in Appendix˜A. To balance the cost and performance of pipeline creation, we select $\bm{\mathcal{M}}$ $\leftarrow$ gpt-4.1-mini whenever an LLM is used to generate or filter data. We discuss model alternatives in Appendix˜B and present a cost breakdown of different model $\bm{\mathcal{M}}$ alternatives for the data pipeline in Table˜10.

2.1 Extract Top Venues from Research Fields

To focus on high-quality surveys, we identify the top-20 publishing venues ranked by h5-index for each field in Google Scholar. Fields on Google Scholar can be overly specific (e.g., Wood Science & Technology) or general (e.g., Health & Medical Sciences (general)), so we manually redistribute from 257 to 94 fields. This process results in 660 venues that are used to retrieve survey articles.

2.2 Extract Survey Articles

We retrieve survey articles using keyword search on three sources: Crossref,111https://www.crossref.org/ Semantic Scholar,222https://www.semanticscholar.org and S2ORC s2orc. In total, 615K candidate articles are returned, 134K of which are downloadable full-text articles, and 54K which are automatically classified to represent a true survey article versus merely containing survey-related keyword(s) in the title.

Article search.

For each of 660 unique venues from Google Scholar, we retrieve publications with title keywords: survey, literature review, a review, an overview, and meta-analysis.

Article filters.

To optimize precision in our search, we apply filters to remove erroneous articles (e.g., survey can be used in the context of field observations, not literature reviews). We prompt $\bm{\mathcal{M}}$ to classify articles as literature reviews (F1 $=$ .80, Prec $=$ .87), removing those judged not to be a literature review, i.e., work aimed at synthesizing and reviewing existing literature.

2.3 Generate Queries from Survey Content

We identify medium-length survey sections with multiple citations, then generate questions and reference answers using grouped sentences from each section. Queries are filtered by keywords and $\bm{\mathcal{M}}$ to ensure they are standalone and without excessive variability in appropriate answers (F1=.86, Prec=.75). In total, 319K of 886K sections are used, yielding 21K queries after filtering.

Survey section filters.

Sections are removed if section titles suggest they are not part of the main body (e.g., abstract). Next, we remove sections too short to be informative ( $<$ 3 sentences or $<$ 800 characters), too long ( $>$ 300K characters), or lacking citations ( $<$ 3 inline citations).

Query and reference answer generation.

We few-shot prompt $\bm{\mathcal{M}}$ to extract hierarchical summaries christensen2014hierarchical from section content. Each summary consists of tree-based structure of questions with supporting sentences found in the section content benz2017questions; wu2023qudeval, helping to generate queries that integrate multiple sources of supporting information (Figure˜7). We prompt $\bm{\mathcal{M}}$ with examples, using section content and the summary to generate a query and reference answer. Prompts instruct that queries must be supported by at least 3 sentences and reference answers summarize supporting sentences without introducing new evidence. Like jansen2025matter, each query is paired with a knowledge cut-off date of the source article.

Query filters.

We aim for queries that are (1) standalone, i.e., understandable by experts without extra decontextualization choi2021decontextualization; and (2) low in answer variability, i.e., different experts are likely to provide similar ratings so that model responses can be more easily scored. To enforce these criteria, we use $\bm{\mathcal{M}}$ to score standaloneness and answer variability of each query on a scale from 1 to 10. Queries scoring $<$ 7 for standaloneness or $>$ 4 for answer variability are removed.333Thresholds validated by NLP experts on initial dev. data. Further, we discard queries that have keywords indicating context dependence (e.g., the paper, this study). We then remove queries with reference answers that are too short ( $<$ 800 characters). Finally, we re-apply $\bm{\mathcal{M}}$ to remove queries that are misassigned to a field, which can occur when mining queries from multi-field venues.

2.4 Rubric Generation

Rubrics $\mathbf{R}=\{R_{i}\}_{i=1}^{K}$ consist of $K$ automatically generated rubric items $R_{i}$ that each evaluate aspects of answer quality. Below, we present the design choices and method for rubric creation.

Desiderata.

Rubrics are distilled from survey articles, adopting limitations from the articles themselves. For example, citations and analysis in the survey article can be subjective. Consequently, there may be gaps in evaluation criteria, e.g., an important missing $R_{i}$ , and not all rubric items may be important. We address these potential limitations through two design choices. (1) We generate and evaluate three rubric types: survey rubrics ( $\mathbf{R}_{S}$ ), parametric rubrics ( $\mathbf{R}_{P}$ ), and hybrid rubrics that merge the two ( $\mathbf{R}_{H}$ $\subset$ ( $\mathbf{R}_{S}$ $\cup$ $\mathbf{R}_{P}$ )), covering obvious gaps in rubrics wadhwa2025evalagent. (2) To create $\mathbf{R}_{H}$ , we deduplicate, rerank, and remove hallucinations from the union set $\mathbf{R}_{S}$ $\cup$ $\mathbf{R}_{P}$ , which intends to remove unimportant or flawed rubric items. The top-8 hybrid rubric items are retained, leaving $\sim$ 7.5 rubric items on average.

Rubric item generation.

We few-shot prompt $\bm{\mathcal{M}}$ , using question-answer pairs from ScholarQABench openscholar, to generate rubric items. $\mathbf{R}_{S}$ is conditioned on both the query and reference answer; $\mathbf{R}_{P}$ is conditioned on only the query. Each rubric item is created from one of three prompts designed to increase diversity of rubric item topics: information-based, depth-based, and citation-based items. Information-based items ask for specific statements, findings, opinions, or comparisons. Depth-based items ask for elaboration or explanation. Citation-based items ask whether answers cite a specific article. To create $\mathbf{R}_{S}$ and $\mathbf{R}_{P}$ with diverse items, we sample 4 information-based items, 2 depth-based items, and 2 citation-based items.

Hybrid rubric construction.

Using $\bm{\mathcal{M}}$ , we deduplicate repeating rubric items that represent identical criteria and rerank rubric items based on their importance for answer evaluation. To remove hallucinations, we remove rubric items referencing papers that cannot be automatically matched to a Google Scholar article. Post-filters, about 61% of hybrid rubric items come from $\mathbf{R}_{S}$ .

Baseline rubric construction.

We additionally compare against a generic rubric ( $\mathbf{R}_{G}$ ) that does not take query specifics into account. Rubric items are sampled from ScholarQABench openscholar and SciArena sciarena metrics: Correctness, Citations, Coverage, Relevance, Organization, Usefulness, Attribution, and Examples. Table˜11 presents the full list of rubric items.

2.5 Dataset statistics and splits

We create train ( $\mathcal{D}_{\text{train}}$ ), validation ( $\mathcal{D}_{\text{validation}}$ ), and test ( $\mathcal{D}_{\text{test}}$ ) splits. $\mathcal{D}_{\text{test}}$ (3.7K queries) samples 50 queries from each field with $\geq$ 50 queries (75 of 94 fields), so that each field is sufficiently represented. $\mathcal{D}_{\text{validation}}$ (703 queries) samples up to 10 queries from remaining fields (74 of 75), and $\mathcal{D}_{\text{train}}$ is made up by the remaining 16.9K queries, which are for supporting the community in developing and tuning research systems.

3 Expert Validation

This section presents the method and results of the expert validation study. We collect expert annotations for three judgement types: query quality (Section˜3.1), rubric quality (Section˜3.2), and scoring and preferences on system outputs (Section˜3.3). Judgements are made on Likert-scale and multi-category labels, visualized in Figure˜3, and are binarized by meaning for downstream analysis.

Annotator recruitment.

We recruited annotators via Ph.D. email lists. 45 Ph.D. students and 1 postdoctoral staff registered to participate in annotations, and 31 ultimately completed the annotation task. Annotators’ expertise span diverse fields: Natural Language Processing (15 experts), Computer Vision (6), Biomedicine (4), Linguistics (3), Physics (2), Genetics (1), Economics (1), and Psychology (1). Annotators were compensated $25 per hour of annotation.

Annotation agreement.

We measure inter-annotator agreement (IAA) as binary pairwise agreement (PA) in Table˜2. While tasks can be subjective, IAA is similar to prior work exploring expert judgements for scientific QA (e.g., pairwise preference is .69 vs .70 PA in openscholar).

3.1 Query Quality

Setup.

Queries are judged on clarity, naturalness, relevance, and variability in valid answers. We compare queries from our pipeline (content-based) against an ablated pipeline that generates solely from survey titles (title-based), controlling for comparisons on the same query topic. This analysis isolates the effect of pipeline components used to enhance query generation and filtering.

Label binarization.

We binarize labels by positive and negative meaning. Clarity merges Largely explicit and Explicit; Naturalness merges Slightly likely to Very likely; Relevance merges Slightly likely to Very likely; and Variability in valid answers merges Semi-open and Closed.

Survey content yields unambiguous queries, eliciting answers without too much variability.

Experts rate $\sim$ 85% of content-based queries as Largely explicit or Explicit, indicating that queries generated from our pipeline have mostly specific and easy interpretations. Likely, content-generated queries have specific interpretations due to grounding in a reference text. Additionally, content-based queries have more Semi-open or Closed-ended queries (96% versus 89%), which supports valid answers that can be directly compared. By contrast, Open-ended queries might elicit a large number of possible answers. For example, “What distinguishes robustness under distribution shift from domain adaptation and transfer learning in NLP?” can yield answers that experts find difficult to rank and compare.

Queries support the information needs of researchers and are naturally expressed.

Queries generated from both full and ablated pipelines have similar relevance and naturalness. 94-90% are rated as Slightly likely or stronger to reflect researchers’ information needs, and 86% of queries Slightly likely or stronger to be expressed in that way by a researcher. These ratings also indicate room for improvement, because queries rarely are rated as Very likely to be relevant or naturally expressed.

3.2 Rubric Quality

Setup.

Rubric items are judged on the number of sentences that should be used to address the item in an answer and how its omission affects answer quality (Figure˜3). Additionally, annotators can flag a number of errors of a rubric item $R_{i}$ : (1) $R_{i}$ is difficult to judge; (2) $R_{i}$ is unclear; (3) $R_{i}$ is an empty or unspecific assessment (e.g., it is a mere rephrasing of the query);444A rule-based system is also employed to detect unspecific rubric items, where rubric items need to include at least one substantive word not present in the original query. (4) $R_{i}$ contains an error (e.g., it cites a non-existent paper). The presence of any of these flags voids the quality of the rubric item. We compare among hybrid, survey, and parametric rubrics.

Label binarization.

In “Proportion of answer measurement”, we merge None and Shorter than a sentence. In “Omission based change in answer quality” we merge Significant improvement to Minor improvement.

Hybrid rubrics are likely to contain criteria worth describing in multiple sentences and improve answer quality.

Across rubric types, rubric items should be described in A sentence or more 84-86% of the time and their omission would degrade answer quality 69-74% of the time. Hybrid rubrics items are rated to be greatly important at a higher frequency, specifically those causing Moderate degradation or Significant degradation to answer quality when removed.

Hybrid rubrics contain few non-existent papers and vacuous restatements of the query.

Parametric rubrics have the highest rate of vacuous, error, or unclear items (15%). Experts note that parametric rubric items contain nonexistent paper titles, causing statements about their coverage in answers to be unanswerable or ambiguous.

3.3 System Output Preferences and Evaluation Protocol Validation

Setup.

We collect expert judgements on head-to-head answers (Figure˜8) to perform a meta-evaluation on LLMs using rubrics to approximate expert judgements. Experts provide two types of annotations: (1) Rubric coverage: Experts rate how well each answer covers each rubric item on a 5-point scale (0 = Not at all, 4 = Completely). (2) Pairwise preference: Experts compare two answers side-by-side (in random order) and select: Left is better, Right is better, Tie, or Both bad. Majority voting determines the final label.

Answers are generated from gpt-4.1-mini and gemini-2.5-flash using their providers’ embedding models555text-embedding-3-large, text-embedding-004 to retrieve relevant passages as input context. We retrieve the top-20 arXiv papers via Google search on the given query and constrain search by the appropriate date cutoff. Papers are chunked into 1000 character passages, and the top-20 passages ranked by embedding similarity score are input as context.

Label binarization.

Rubric coverage merges (0 = Not at all, 1 = Barely) and (2 = Moderately, 3 = Mostly, 4 = Completely). For pairwise preference, we drop Tie and Both bad, keeping only direction labels, i.e., Left or Right. PA are 0.73 and 0.69 respectively. Because agreement for pairwise preference is low, we collect up to 4 expert judgements and keep only those that had a majority voted direction label (89.7% of cases with direction judgements). Human annotators with direction label preferences have 84% accuracy agreement with the majority vote label.

Evaluator LLMs.

Open-source evaluators (prometheus-2-8x7b; prometheus-two) and proprietary LLMs (claude-4-sonnet and gpt-4.1-mini) are validated as automatic evaluators. For each evaluator, we compute agreement to expert annotations on rubric coverage and pairwise preference, which represent absolute and relative measures of answer quality respectively.

We prompt an LLM to produce rubric coverage in one call, inputting a system answer $A$ and rubric $\mathbf{R}=\{R_{i}\}^{K}_{i=1}$ to quantify the number of rubric items covered in $A$ on a 5-point scale, $\mathrm{Coverage}:A\times\mathbf{R}\rightarrow\{0,1,2,3,4\}^{K}$ . The prompt describes the ends of the scale with labels from the user study (0 = Not at all, 4 = Completely).

We compare two types of automatic judges to predict pairwise preference. The Direct Judge ( $\mathrm{J}_{\text{direct}}$ ) is prompted with the query and two system answers and asked “Which response is better?”. Both answer orderings are evaluated to reduce positional bias shi2024judging. However, direct comparison may lack appropriate knowledge useful to predict expert preferences. To address this gap, we introduce Ensemble Judge ( $\mathrm{J}_{\text{ensemble}}$ ), which uses both $\mathrm{J}_{\text{direct}}$ and $\mathrm{Coverage}$ to predict preferred answers: the answer with a larger sum of $\mathrm{Coverage}$ (0-4 scale) and a full 4 points for each $\mathrm{J}_{\text{direct}}$ comparison, is marked as the preferred answer. Formally, this sum can be expressed as:

[TABLE]

where $\mathbb{I}_{A}$ indicates 1 if $A$ is preferred by $\mathrm{J}_{\text{direct}}$ and 0 otherwise; $\mathbb{I}^{\prime}_{A}$ implements $\mathrm{J}_{\text{direct}}$ with answers input in the reverse order. $\mathrm{J}_{\text{ensemble}}$ prefers the answer with the larger sum or outputs a tie.

LLM-Human agreement on Rubric Coverage.

All evaluator LLMs show fair Pearson correlation with expert annotations on rubric coverage: prometheus-2-8x7b at .48, and both claude-4-sonnet and gpt-4.1-mini at .63. Averaged over our annotated data, experts and gpt-4.1-mini differ on $\mathrm{Coverage}$ by only 0.11. On the other hand, gpt-4.1-mini tends to make more extreme predictions of $\mathrm{Coverage}$ for individual samples. Experts rate a rubric items Completely covered 12.3% of the time while gpt-4.1-mini does so 26.0%. While a suitable judge on average, conclusions with gpt-4.1-mini may overestimate the frequency of some coverage values at the extremes.

LLM-Human agreement on Pairwise Preference.

We present the preference ranking accuracies of evaluator LLMs with respect to majority labels in Figure˜4. Each graph presents a base direct judge and the results of ensembling with the four types of rubrics we consider. In cases where judges predict ties, we assign partial credit.666We have 3 cases of partial credit: (1) for the direct judge, if $\mathbb{I}_{A}\neq\mathbb{I}^{\prime}_{A}$ , 50%, (2) if $\mathbb{I}_{A}$ or $\mathbb{I}^{\prime}_{A}$ is correct and the other reports tie, 75%, and (3) if $\mathrm{J}_{\text{ensemble}}$ reports tie, 50%. Human-Human accuracy agreement indicates an upper bound of 84% for LLM-Human agreement. Ensemble judges ( $\mathrm{J}_{\text{ensemble}}$ ) are consistently better estimators of expert preferences than direct judges ( $\mathrm{J}_{\text{direct}}$ ), achieving up to $\sim$ 75% accuracy. Notably, rubrics can decrease the LLM-Human and Human-Human agreement gap from 12.7% to 9.6%, corresponding to a 24% relative reduction. In general, query-specific rubrics match or outperform generic rubrics. Hybrid rubrics demonstrate high performance gain across all systems, allowing prometheus-2-8x7b to match the direct judge performance of claude-4-sonnet. As visualized in Figure˜11, hybrid rubrics enhance evaluation in 7 of 8 fields, where they are the best-performing rubric in 4 of 8. The performance of parametric and generic rubrics depend on the strength of the evaluator: in strong evaluators, they can match hybrid rubrics, but they yield little to no benefit in open evaluators.

4 LLM Systems Evaluation Setup

To benchmark top systems used for scholarly inquiry on ResearchQA, we evaluate 18 parametric, retrieval augmented, and deep research systems in both open-source and proprietary families (Table˜3) on $\mathcal{D}_{\text{test}}$ . We perform all analyses with the best-performing rubrics (hybrid rubrics), evaluator LLM (gpt-4.1-mini), and protocol ( $\mathrm{J}_{\text{ensemble}}$ ). We describe the task setup in Section˜4.1 and tournament setup in Section˜4.2.

4.1 Evaluation Task and Metrics

Task.

We consider the following task: systems are input a query $Q$ and generate a citation-supported answer $A$ , constrainting on date $D$ and a response length $L=\,$ 250 words. A sample answer is visualized in Figure˜9. This task design conservatively guards against potential biases from information recency, multi-turn clarification, and length biases rlhflengthbiases, trading off task naturalness. We explore more natural setups in section˜5.2, finding that $D$ can be presently discarded with low likelihood of bias. Further, we remove $L$ , allowing longer responses that may result from multi-turn interaction or no length specification.

Coverage %.

We compute average $\mathrm{Coverage}$ over answers and divide by 4, normalizing the resulting percentage from 0% to 100%.

Leaderboard score.

Consistent with Chatbot Arena chatbotarena, we use the Bradley-Terry model bradley-terry-model for pairwise battles judged by $\mathrm{J}_{\text{ensemble}}$ , which is equivalent to the Elo equation elo. In case of ties, each system wins half the match. We report the median score from a 1000-iteration bootstrap and its standard deviation.

4.2 Tournament Details

We explore systems at a gradation of engineering effort: parametric, naive retrieval, production retrieval, and deep research.

Parametric systems.

We directly generate answers using default configs of API providers (OpenRouter777https://openrouter.ai/ for open-source models).

Naive retrieval systems.

These systems represent retrieval with minimal engineering effort: language models conditioned on the top-20 retrieved passages. Each language model is paired with an embedding model from the same provider. We retrieve papers from a date-constrained Google Scholar search. For each query, up to 50 papers are retrieved: the top-20 using the query as search field, and an additional 10 for each of 3 related keywords generated by gpt-4.1-mini, a search method used in openscholar. Papers corresponding to distilled survey papers are removed to prevent unfair evaluation advantages due to data leakage. Each paper is chunked into 1000-token passages with 200 overlapping tokens of subsequent chunks, and the top-20 passages by embedding similarity score are used as context, optionally reranking when a cross-encoder reranker is available. We generate answers using the same decoding configurations as in the parametric setting and employ greedy decoding for OpenScholar.

Production retrieval systems.

These systems represent retrieval with advanced engineering effort, e.g., tool use, feedback, or refinement. We use default implementations in their respective repositories or APIs without modification.

Deep research systems.

At the time of writing, two commercial deep research systems have API access: OpenAI and Perplexity deep research. We use both APIs without modification.

Battle construction.

We construct a total of 7.6K battles, where each domain consists of a minimum of 1K battles and up to as many queries represented in the domain (1.8K for Health Sciences & Medicine). Each battle is constructed such that models are sampled uniformly ( $\sim$ 800 battles per model) and all model matchups are uniformly represented in each domain, ensuring connectivity of the comparison network.

5 LLM Systems Evaluation Results

5.1 Response Results

Results by systems.

All systems struggle to fully cover rubric items, with no parametric or retrieval augmented systems we evaluated exceeding 70% coverage (Table˜3). The best-performing system sonar-deep-research reaches not much more, obtaining 75% coverage. These results demonstrate headroom for further improvement. Win-rates of different systems indicate large performance gaps: systems designed for research synthesis have large advantages over others. In fact, sonar-deep-research is estimated to have an 82% win rate over the next highest rated system, gemini-2.5-pro.

We visualize sample responses from each system category in Figure˜9 and Figure˜10, presenting a case study to show qualitative differences. Here, the question is: “What strategies are recent research efforts employing to reduce the wall-clock training time for BERT models? (Date: 2020-10-02)”. The parametric response covers sensible methods for training optimization (mixed-precision training, gradient accumulation) addressing content mentioned in rubrics. However, the response lacks detail about each method. Naive retrieval locates and explains in detail trainbigcompress, describing that the most effective recent strategies involve training larger models for fewer steps. The response hardly addresses prototypical explanations to the query, scoring lower coverage on the rubric. We further compare production retrieval and deep research for the query, “How does substrate temperature influence the crystal orientation and quality of AlN (002) thin films during sputtering? (Date: 2020-10-19)”. While the production retrieval response appears sensible, the deep research answer mentions the same explanations (atomic mobility) while uniquely discussing further aspects of temperature influences (stress evolution) in detail (<300°C [increases] residual stress up to 1.2GPa). In general, we find that deep research systems cover prototypical explanations while also dense in breadth and depth, demonstrating several advantages over systems from other categories.

Results by rubric type.

To identify areas for improvement, we break down coverage of rubric items addressing different types of evaluation criteria. Each rubric item is lemmatized, using NLTK nltk, and categorized using rules to 6 rubric types (Macro F1 $=$ .92, without “Other”). In Table˜5, we show examples and the distribution of sonar-deep-research errors. We report the proportion of rubric item types that are not Completely covered: citation-based items can be improved the most (89%), followed by describing important limitations (52%) and making comparisons (52%). Error breakdowns of all other systems are shown in Figures˜12 and 13.

Results by domains.

Differences between domains are within $\pm$ 6 Coverage % (Table˜3). While differences are small, Health Sciences and Humanities rank as the 2 lowest performing domains: in 10/18 systems for Health Sciences and 18/18 for Humanities. These trends are suggestive that evaluations focused on Engineering and Physical Sciences, which rank as top-performing domains and are also heavily represented in prior evaluations (Table˜1), provide an incomplete analysis of performance across research domains.

5.2 Effects of Information Recency, Survey Leakage, and Answer Length

In this section, we explore biases on Coverage % due to citing recent information, survey retrieval leakage, and constrained answer lengths.

Information recency analysis.

We examine whether relying on sources published after the distilled survey articles unfairly lowers Coverage %. For each answer, we count the % of cited sources that violate the instructed date cutoff (survey article publication year). Sources that violate the date cutoff make up $\sim$ 30% of the citations (Figure˜5), suggesting systems do not adhere to date cutoffs stated in instruction text. Despite high frequency of date violations, we observe that recent sources do not positively or negatively affect Coverage % in aggregate: Coverage % is on average $\pm 1.3$ % relative to mean coverage when there are no date violations. While current effects of date violations are small, we recommend monitoring possible information recency biases in future evaluations.

Leakage analysis.

By distilling rubrics from downloadable survey papers, systems may gain evaluation advantages when retrieving the same survey papers to generate answers, i.e., leakage. To mitigate leakage from affecting evaluation, we restrict source papers from appearing in search retrievals in systems where appropriate control is possible. We additionally perform post-hoc leakage analysis, because many production systems do not have the option to restrict search retrieval for answer generation. Leakage is detected by checking whether the survey titles appears in the reference section of the answer.888To address noise in paper titles, we lowercase text, remove special characters, and search for 6-gram matches. We compare Coverage % in subsets of answers where leakage is and is not detected for top-performing production systems in Table˜6 and all systems in Table˜13.

Leakage advantages are small and inconsistent: in 13 systems exhibiting leakage, Coverage % increased for answers with leakage in 8/13 systems, where Coverage % actually decreased in the other 5/13 systems. In aggregate, we observe increases of $\sim$ 1.1% when distilled surveys are retrieved to generate answers. These small differences assuage our concerns of leakage advantages in current systems, but evaluations of future systems should continue verifying that leakage advantages are minimal and restrict source surveys from search results whenever possible.

Unconstrained answer length analysis.

We remove the 250 word constraint from the task instruction text to further explore how Coverage %, a recall-based metric, increases when we do not control for answer length during evaluations. We select from the top performers for parametric, naive retrieval, and deep research systems by Coverage % and additionally generate answers for 3 queries per field (225 total queries). Average statistics are shown in Table˜4, and trends for deep research systems are visualized in Figure˜6 and for the remaining systems in Figure˜15.

Unconstrained length analyses reveal that many systems naturally generate answers of close to 250 words, with no substantial differences to answer length or Coverage % (Figure˜15). However, deep research systems tend to generate longer answers, up to 3K words (about 6 pages of text) per answer. By contrast, expert-written answers lengths for similar topics average 289 words in openscholar. Coverage % increases with answer length up to $\sim$ 2K words (about 4 pages of text), as shown in Table˜4, plateauing at 85% for sonar-deep-research and 79% for o4-mini-deep-research. These trends suggest that deep research systems are able to convey more helpful information while trading off concise and focused answers.

6 Related Work

We fit into a broad body of work trying to improve benchmarking for LLMs. This includes multi-domain works hendrycks2020measuring; wang2025mmlu, challenge sets rein2024gpqa; phan2025humanity; wolfson2025monaco and expert annotations expertqa; malaviya2025dolomites. Model development for survey creation has used survey data material goldfarb2020scalelitreview; kasanishi2023scireviewgen; agarwal2024litllm; wang2024autosurvey. Below we outline the key differences between ResearchQA and others.

Manually crafted scholarly benchmarks.

Manual curation of benchmarks for scholarly QA has encountered practical challenges for creating large, diverse, and complex datasets. QASPER qasper and QASA qasa limit their focus on queries that can be answered within a single paper. KIWI kiwi is built from questions derived by researchers on related work sections, and ScholarQABench openscholar recruited researchers to write questions from scratch. Both efforts have multi-document queries but are smaller than ResearchQA because of the challenges of recruiting researchers. Concurrently, SciArena sciarena leverages community contributions to collect queries and preferences; however, it remains smaller in size and coverage than ResearchQA and lacks evaluation rubrics.

Auto-generated scholarly benchmarks.

Automatically generated scholarly QA benchmarks trade-off scale for complexity and naturalness. DeepScholar-Bench patel2025deepscholar uses 63 CS papers to compare machine-written and human-written related works. SciQA sciqa is template generated and focuses on questions generated from knowledge graphs. PubMedQA pubmedqa generates queries from the abstracts of PubMed articles, but limits to yes or no questions. SciDQA scidqa extracts 188k queries asked during peer review on OpenReview but limits queries to extractions about the paper being reviewed. ResearchQA achieves more abstractive queries and evaluation materials by focusing on surveys but is smaller than SciDQA.

Evaluating long-form answers and rubrics.

A central problem for long-form answer evaluation is a large space of possible correct answers. Reference answers can be difficult to use, and token based measures like ROUGE lin2004rouge are gameable krishna2021hurdles. Recent efforts rely on direct evaluator LLMs openscholar; alpacaeval but may inherit self-preference llm-judge-self-bias-panickssery; llm-judge-self-bias-watoka, length biases alpacaeval and may be inaccurate for research queries. ResearchQA shows these judges can be effective. Rubric-based evaluations that decompose judgment into nuanced criteria sawadaarb; g-eval are promising alternatives to direct evaluator LLMs. Manually curated rubrics are often task-specific and small scale openscholar; starace2025paperbench; qin2024infobench. WildBench lin2024wildbench creates query-specific checklist rubrics from parametric memory at scale. EvalAgent wadhwa2025evalagent discovers query-specific rubrics with online search. Our rubrics leverage discovery from surveys, aligning them to expert-written material.

7 Conclusions

We introduce ResearchQA, a resource distilled from survey articles for large scale and multi-field evaluations of research synthesis. We leverage ResearchQA to benchmark 18 systems showing each has headroom for improvement. The highest-ranking deep research system, which achieves 82% win rate over the next system, only fully addresses fewer than 11% of items addressing citations, 48% of items describing limitations, and 49% of items asking about comparisons. ResearchQA can be expanded with newly written surveys as to increase coverage of new topics.

Limitations

While ResearchQA reduces reliance on experts, expert involvement is not eliminated. We depend on experts and expert-written articles to build and validate ResearchQA. To the extent that we can, we recruit experts to validate data, but not all fields are equally validated. Additionally, open access to research articles is limited, hindering exhaustive coverage of the literature. Rubrics distilled from these articles may not fully cover important criteria; we leave synthesis of larger and comprehensive rubrics for future work. Many recently released deep research systems involve limited access; their omission can distort leaderboard rankings and obfuscate important failure modes.

Acknowledgements

This research was developed with funding from the Defense Advanced Research Projects Agency’s (DARPA) SciFy program (Agreement No. HR00112520300) and the National Science Foundation Graduate Research Fellowship under Grant No. DGE-2236662. The views expressed are those of the authors and do not reflect the official policy or position of the sponsors. We would like to thank the UPenn NLP group and Material Science and Engineering, Biomedical Engineering, Computer and Information Science, Genetics, Linguistics, Physics and Astronomy, Criminology, and Psychology departments for generously participating in our annotation. We thank Zack Ives for aiding university wide recruitment.

Appendix A Additional Details of Dataset Pipeline

Below, we describe the data pipeline (Section˜2) details.

A.1 Query generation from survey articles

Curating a list of top publication venues for 257 research fields.

We identify the top-20 publishing venues for each of 257 research fields listed on Google Scholar Metrics (total venues $=$ 4634). The aggregate counts are presented in Table˜7.

Retrieving survey articles from each publication venue.

We retrieve articles from three datastores: Crossref, the Semantic Scholar API, and S2ORC s2orc. Articles are queried with the following keywords: survey, literature review, a review, an overview, and meta-analysis. The article retrieval yields 615K article weblinks, where 134K are downloadable full-text articles.

Removing non-survey articles.

We use $\bm{\mathcal{M}}$ to classify between actual literature reviews and articles that mention a keyword like survey in its title. Classifier metrics are F1 $=$ .80, Prec $=$ .87, Rec $=$ .75 on an author-annotated validation set. The final yield is 54K survey articles across 254 research fields (3 research fields do not yield survey articles using our method).

Selecting sections from survey articles.

We apply a sequence of filters to select sections (total $=$ 886K, yield $=$ 319K) from survey articles:

•

Title passes keyword blacklist: We ignore abstract, introduction, or other summary sections, removing any sections that contain the following words: question, survey, abstract, introduction, contribution, related, result, discussion, conclusion, limitation, appendix, appendices, appendixes, supplementary, supplemental, supplement, material, acknowledgement, future, direction, summary, suggestion, table, tab., tbl., figure, fig., and plot.

•

Section length is not too short ( $\geq$ 3 sentences, $\geq$ 800 characters) and not too long ( $\leq$ 300K characters): We apply a basic length filter to ensure that queries are generated from substantial, but focused, sections.

•

Minimum number of in-text citations ( $\geq$ 3): We select sections that are well grounded in the literature, where each section must have 3 or more citations.

Generating queries from survey sections.

We extract a date cutoff $D$ from each section’s article metadata, and we use $\bm{\mathcal{M}}$ to parse the section content into a hierarchical summary to generate an initial query and initial reference answer $(Q_{\text{initial}},\hat{A}_{\text{initial}})$ . To select high quality queries, all queries (total $=$ 319K, yield $=$ 21K) are filtered:

•

Initial query is self-contained: $\bm{\mathcal{M}}$ assigns $Q_{\text{initial}}$ a self-containment score from 1 to 10, where higher scores indicate more self-containment. We remove queries with $<$ 7.

•

Initial query has low answer variability: $\bm{\mathcal{M}}$ assigns $Q_{\text{initial}}$ an answer variability score from 1 to 10, where higher scores indicate likely answer variability (e.g., expert disagreement or subjectivity). We remove queries with $>$ 4.

•

(Rephrasing Step) Query and reference answer cohesion: $\bm{\mathcal{M}}$ rephrases $(Q_{\text{initial}},\hat{A}_{\text{initial}})$ to queries $Q$ and reference answers $\hat{A}$ to improve their cohesion. We do not remove any queries at this step.

•

Query does not contain a citation: $\bm{\mathcal{M}}$ detects whether a citation is in the query $Q$ .

•

Reference answer length is long enough ( $\geq$ 800 characters): Reference answers $\hat{A}$ substantiate a semi open-ended query.

•

Final query is self-contained: After rephrasing, some queries are still not standalone, so we apply a keyword-based method to remove non-standalone queries. Keywords include past tense auxiliary verbs (did, was, were) or words in referring expressions (questionnaire, literature).

•

Final query is aligned with field: Venues can be multidisciplinary, and therefore some queries are unrelated to the field those venues are mapped to. To ensure field alignment, $\bm{\mathcal{M}}$ clasifies and removes queries that are unrelated to the field.

•

Queries are from a field with $\geq$ 50 queries: Queries should cover important aspects of a field, which is difficult to accomplish if there are not enough queries. We set 50 to be the minimum query count to call a field sufficiently covered by queries, which leaves 21K queries $Q$ that cover 75 research fields.

A.2 Rubric generation from survey articles

We create rubrics with $\bm{\mathcal{M}}$ . To diversify rubric item types, three generation prompts are used—information, depth, and citation. These rubric types are intended to diversify the types of rubric items that are generated, whereas rubric items have been categorized post-hoc to more granular labels for analysis.

•

Information-based rubric item: A binary yes/no question asking whether an answer addresses a specific statement, finding, opinion, or comparison. Rubric items of this category allow for broad analysis of quality.

Example: Does the response address the specific benchmarks where auto-regressive LMs outperform bi-directional encoders?

•

Depth-based rubric item: A binary yes/no question asking whether an answer elaborates or explains a topic. Rubric items of this category allow measuring analysis, discussion, and explanation in answers.

Example: Does the response elaborate on the hybrid approach of GRIT in unifying auto-regressive and bi-directional features?

•

Citation-based rubric item: A binary yes/no question asking whether or not an answer cites a specific study. Rubric items of this category evaluate answer grounding in the literature.

Example: Does the response cite papers such as Wang et al. (2023) (title: Improving text embeddings with large language models) and the MTEB paper (title: Massive Text Embedding Benchmark) that show the performance of auto-regressive LMs surpassing bi-directional encoders in retrieval tasks?

In addition to automatically generated rubrics, we explore generic rubrics using evaluation criteria from related work openscholar; sciarena. Generic rubric items are in Table˜11.

Remapping research fields and domains.

For the purposes of evaluation, some fields are overly specific (e.g., Wood Science & Technology) or overly general (e.g., Health & Medical Sciences (general)). Additionally, the Physics & Mathematics branch does not yield enough queries to sufficiently represent a domain. For these reasons, we redistribute data to match a more intuitive hierarchical structure. In total, we merge 170 of 257 fields and introduce 7 more: Animal Biology & Behavior, Biomedical Engineering, Electrical Engineering, Management, Mathematics, Physics, and Neuroscience. These changes result in a hierarchy covering 94 fields and 7 domains. Our pipeline is able to sufficiently cover 75 of 94 fields (where each of the 75 fields has $\geq$ 50 queries) using the available datastores. A full list of fields is in tabulated in Table˜8.

Appendix B Pipeline Model Analysis

In this section, we discuss the performances and potential costs of different models $\bm{\mathcal{M}}$ used in the pipeline to create ResearchQA. We consider three state-of-the-art models at the time of consideration in Table˜10.

$\bm{\mathcal{M}}$ as a classifier.

We use $\bm{\mathcal{M}}$ as a classifier for quality filtering, such as removing non-literature reviews, ambiguous queries, open-ended queries, and removing queries with citations. For each filtering task, we find that the smallest model, gpt-4.1-mini, obtains at least .80 F1 compared against author annotations.

$\bm{\mathcal{M}}$ as a generator.

We show sample outputs from using gpt-4.1-mini and gpt-4.1 in Table˜9. We find that queries and rubrics generated by the two models are comparable in quality and content. Queries generated by gpt-4.1 are more specific, such as asking explicitly how linguistics and dictionaries are applied in rule-based machine translation; by contrast, gpt-4.1-mini asks for an open-ended explanation of rule-based machine translation with benefits and limitations. These differences propagate to the rubrics, with those from gpt-4.1 judging by discussion of word ambiguities and parts-of-speech whereas those from gpt-4.1-mini judge by data efficiency in rule-based methods ( $R_{3}$ ). Overall, we find content similarities between model outputs with small differences in the fine-grained evaluation criteria.

Costs.

Costs for batched inference of $\bm{\mathcal{M}}$ are in Table˜10. Notably, gpt-4.1 is $\sim$$2K more expensive than gpt-4.1-mini. Given performance similarities, we choose gpt-4.1-mini for scaling out ResearchQA.

Appendix C Model Prompts

C.1 Survey filtering prompt: is literature review

System: Determine if the academic paper title indicates a LITERATURE REVIEW.

Answer ONLY “Yes” if the paper’s PRIMARY PURPOSE is to analyze, synthesize, and evaluate research on a topic.

Answer ONLY “No” for all other paper types, including:

Papers describing user studies, surveys of people, or field observations
Papers that report on events, occurrences, or observations (even if they use “review” in the title)
Reports that summarize disease outbreaks, statistics, or time-period data
Papers that evaluate or compare methods for conducting literature reviews

IMPORTANT DISTINCTIONS:

A LITERATURE REVIEW is about analyzing research itself
Having words like “review,” “survey,” “meta-analysis,” or “overview” in the title is NOT sufficient

User: Paper Title: ** <PAPER_TITLE>**

C.2 Query generation prompt: hierarchical summarization

System: In this task, you will be given a section from an academic paper, split into individual sentences.

You will also receive a set of questions based on a partial section in a hierarchical manner, where each question is clearly answered by the sentence indicated by its index.

You need to generate a question that, within the scope of the hierarchical question set, requires understanding the corresponding part of this section.

Ideally, this question should be based on the sentence, along with sufficient following sentences, as indicated in the hierarchical question set.

Also, provide the supporting sentence indices.

Content

Paper Title: Societal Biases in Language Generation: Progress and Challenges

Section Title: Bias Definitions and Metrics

Content:

sent1: In the context of AI fairness, the term “bias” commonly refers to skews that result in undesirable impacts (Crawford, 2017) and is quantifiable with some metric.

sent2: There are relatively more existing studies on biases in NLU tasks, where it is arguably simpler to define bias metrics, since we can intuitively compare the accuracy of the task (e.g., coreference resolution, hate speech detection) for different demographics.

sent3: Language generation tasks often involve stochastic generation of open-ended and lengthy texts, traits that are not directly compatible with traditional algorithmic bias definitions (e.g., equalized odds, equal opportunity, demographic parity (Dwork et al., 2012; Hardt et al., 2016)).

sent4: Because of the difficulty in defining metrics, existing works define bias loosely as demographic inequality and use intermediate proxy metrics to comparatively measure bias.

sent5: Examples include: Regard Ratio: negative-neutral-positive regard score ratios of text generated from bias-inducing prompts (Sheng et al., 2019).

sent6: Sentiment Ratio: negative-neutral-positive sentiment score ratios of text generated from African American English (AAE) versus White-Aligned English (WAE) prompts (Groenwold et al., 2020).

sent7: Individual and Group Fairness through Sentiment: comparisons of the sentiment distributions of generated text across demographics and prompts (Huang et al., 2020).

sent8: Gendered Word Co-occurrence Score: mean and standard deviations of the absolute log ratio of probabilities: P(word|female terms) to P(word|male terms) across all words in generated text (Bordia and Bowman, 2019).

sent9: There are also metrics for other bias evaluation setups in continuation generation tasks involving sentiment (Shwartz et al., 2020), the ratio of gendered words (Solaiman et al., 2019; Vig et al., 2020; Dinan et al., 2020a), and other novel metrics (Peng et al., 2020; Yeo and Chen, 2020).

sent10: Studies of biases in transformation generation tasks favor metrics of accuracy in terms of successfully transforming text to have a desired property.

sent11: We present a more thorough comparison of metrics in Section 5.4.

sent12: Bias metrics can also be categorized by how they define associations between demographic group attributes and text.

sent13: Biases can be towards people described in text, people who produce the text, or people to whom the text is addressed (Dinan et al., 2020b).

sent14: Most existing works define bias metrics through the first association—these biases are relatively easier to analyze, since both the demographic and the textual signals of bias are encapsulated within the text.

sent15: There are also works that define biases towards people who produce the text (Groenwold et al., 2020) or people to whom the text is addressed (Sheng et al., 2021b), though there are relatively fewer works that study these latter associations.

Output:

What is the common definition of “bias” in the context of AI fairness? sent1
How are biases in NLU tasks typically defined and measured? sent2
Why are traditional algorithmic bias definitions not directly compatible with language generation tasks? sent3
How do existing works define and measure bias in language generation tasks? sent4

4.1. What are some examples of intermediate proxy metrics used to measure bias in language generation tasks? sent5, sent6, sent7, sent8

4.2. Are there other metrics for bias evaluation in continuation generation tasks? sent9

4.3. What metrics are favored in transformation generation tasks? sent10

4.4. Where can a more thorough comparison of metrics be found? sent11

How can bias metrics be categorized? sent12
What are the different associations between demographic group attributes and text in bias metrics? sent13

6.1. Which association is most commonly used in existing works and why? sent14

6.2. Are there works that define biases towards people who produce the text or people to whom the text is addressed? sent15

Content

Paper Title: Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing

Section Title: Hand-Crafted Documentation in Typological Databases

Content:

sent1: Typological databases are created manually by linguists.

sent2: They contain taxonomies of typological features, their possible values, as well as the documentation of feature values for the world’s languages.

sent3: Major typological databases, listed in Table 1, typically organize linguistic information in terms of universal features and language-specific values.

sent4: For example, Figure 3 presents language-specific values for the feature number of grammatical genders for nouns on a world map.

sent5: Note that each language is color-coded according to its value.

sent6: Further examples for each database can be found in the rightmost column of Table 1.

sent7: Some databases store information pertaining to multiple levels of linguistic description.

sent8: These include WALS (Dryer and Haspelmath 2013) and the Atlas of Pidgin and Creole Language Structures (APiCS) (Michaelis et al. 2013).

sent9: Among all presently available databases, WALS has been the most widely used in NLP.

sent10: In this resource, which has 142 typological features in total, features 1–19 deal with phonology, 20–29 with morphology, 30–57 with nominal categories, 58–64 with nominal syntax, 65–80 with verbal categories, 81–97 and 143–144 with word order, 98–121 with simple clauses, 122–128 with complex sentences, 129–138 with the lexicon, and 139–142 with other properties.

sent11: Other databases only cover features related to a specific level of linguistic description.

sent12: For example, both Syntactic Structures of the World’s Languages (SSWL) (Collins and Kayne 2009) and AUTOTYP (Bickel et al. 2017) focus on syntax.

sent13: SSWL features are manually crafted, whereas AUTOTYP features are derived automatically from primary linguistic data using scripts.

sent14: The Valency Patterns Leipzig (ValPaL) (Hartmann, Haspelmath, and Taylor 2013) provides verbs as attributes and predicate–argument structures as their values (including both valency and morphosyntactic constraints).

sent15: For example, in both Mandinka and Sliammon, the verb to laugh has a valency of 1; in other words, it requires only one mandatory argument, the subject.

sent16: In Mandinka the subject precedes the verb, but there is no agreement requirement; in Sliammon, on the other hand, the word order does not matter, but the verb is required to morphologically agree with the subject.

sent17: For phonology, the Phonetics Information Base and Lexicon (PHOIBLE) (Moran, McCloy, and Wright 2014) collates information on segments (binary phonetic features).

sent18: In the Lyon–Albuquerque Phonological Systems Database (LAPSyD) (Maddieson et al. 2013), attributes are articulatory traits, syllabic structures, or tonal systems.

sent19: Finally, StressTyp2 (Goedemans, Heinz, and der Hulst 2014) deals with stress and accent patterns.

sent20: For instance, in Koromfé each word’s first syllable has to be stressed, but not in Cubeo.

sent21: Other databases document various aspects of semantics.

sent22: The World Loanword Database (WOLD) (Haspelmath and Tadmor 2009) documents loanwords by identifying the donor languages and the source words.

sent23: The Automated Similarity Judgment Program (ASJP) (Wichmann, Holman, and Brown 2016) and the Intercontinental Dictionary Series (IDS) (Key and Comrie 2015) indicate how a meaning is lexicalized across languages:

sent24: For example, the concept of WORLD is expressed as mir in Russian, and as ārkiśos. i in Tocharian A.

sent25: Although typological databases store abundant information on many languages, they suffer from shortcomings that limit their usefulness.

sent26: Perhaps the most significant shortcoming of such resources is their limited coverage.

sent27: In fact, feature values are missing for most languages in most databases.

sent28: Other shortcomings are related to feature granularity.

sent29: In particular, most databases fail to account for feature value variation within each language: They report only majority value rather than the full range of possible values and their corresponding frequencies.

sent30: For example, the dominant adjective–noun word order in Italian is adjective before noun; however, the opposite order is also attested.

sent31: The latter information is often missing from typological databases.

sent32: Further challenges are posed by restricted feature applicability and feature hierarchies.

sent33: Firstly, some features apply, by definition, only to subsets of languages that share another feature value.

sent34: For instance, WALS feature 113A documents “Symmetric and Asymmetric Standard Negation,” whereas WALS feature 114A “Subtypes of Asymmetric Standard Negation.”

sent35: Although a special NA value is assigned for symmetric-negation languages in the latter, there are cases where languages without the prerequisite feature are simply omitted from the sample.

sent36: Secondly, features can be partially redundant, and subsume other features.

sent37: For instance, WALS feature 81A “Order of Subject, Object and Verb” encodes the same information as WALS feature 82A “Order of Subject and Verb” and 83A “Order of Object and Verb,” with the addition of the order of subject and object.

Output:

What are typological databases and how are they created? sent1

1.1. What do typological databases contain? sent2

1.2. How is linguistic information organized in major typological databases? sent3

1.2.1. Can you provide an example of how linguistic information is organized? sent4, sent5

1.2.2. Where can further examples for each database be found? sent6

1.3. Do some databases store information at multiple levels of linguistic description? sent7

1.3.1. Which databases store information at multiple levels of linguistic description? sent8

1.3.2. Which database is the most widely used in NLP? sent9

1.3.2.1. What features does WALS cover? sent10

1.4. Do other databases cover specific levels of linguistic description? sent11

1.4.1. Can you provide examples of databases that focus on specific levels of linguistic description? sent12

1.4.1.1. How are features in SSWL and AUTOTYP crafted? sent13

1.4.1.2. What does the Valency Patterns Leipzig (ValPaL) provide? sent14

1.4.1.2.1. Can you provide an example of valency in different languages? sent15, sent16

1.4.1.3. What information does PHOIBLE collate? sent17

1.4.1.4. What attributes are included in LAPSyD? sent18

1.4.1.5. What does StressTyp2 deal with? sent19

1.4.1.5.1. Can you provide an example of stress patterns in different languages? sent20

1.5. Do other databases document aspects of semantics? sent21

1.5.1. What does the World Loanword Database (WOLD) document? sent22

1.5.2. What do ASJP and IDS indicate? sent23

1.5.2.1. Can you provide an example of how a meaning is lexicalized across languages? sent24

What are the shortcomings of typological databases? sent25

2.1. What is the most significant shortcoming? sent26

2.1.1. What is the evidence for the limited coverage of typological databases? sent27

2.2. What are other shortcomings related to? sent28

2.2.1. How do most databases fail to account for feature value variation within each language? sent29

2.2.1.1. Can you provide an example of missing information in typological databases? sent30, sent31

2.3. What further challenges are posed by typological databases? sent32

2.3.1. What is the first challenge related to? sent33

2.3.1.1. Can you provide an example of restricted feature applicability? sent34, sent35

2.3.2. What is the second challenge related to? sent36

2.3.2.1. Can you provide an example of feature redundancy and subsumption? sent37

Content

Paper Title: Efficient Methods for Natural Language Processing: A Survey

Section Title: Sparse Modeling

Content:

sent1: To leverage sparsity for efficiency, many models follow the mixture-of-experts (MoE) concept (Jacobs et al., 1991; Shazeer et al., 2017; Fedus et al., 2022a), which routes computation through small subnetworks instead of passing the input through the entire model.

sent2: Relevant works on this line include GShard (Lepikhin et al., 2021), Switch Transformer (Fedus et al., 2022b), and ST-MoE (Zoph et al., 2022), which replace the feed forward layers in transformers with MoE layers.

sent3: More recently, Rajbhandari et al. (2022) scaled transformers up by compressing and optimizing the usage of MoE.

sent4: Overall, MoE models have been shown to achieve strong performance across several NLP tasks while reducing the overall resource consumption (Sec. 8).

sent5: For instance, GLaM (Du et al., 2022) used only $\sim$ 1/3 of GPT-3’s energy consumption (with additional hardware-based optimization), while Rajbhandari et al. (2022) reached a 5x reduction in terms of training cost.

sent6: However, MoE models have also exhibited training instabilities in practice, and may require architecture-specific implementation (Zoph et al., 2022; Mustafa et al., 2022).

sent7: Another promising direction for exploiting sparse modeling is Sparsefinder (Treviso et al., 2022), which extends the Adaptively Sparse Transformer (Correia et al., 2019) to allow a more efficient attention mechanism by identifying beforehand the sparsity pattern returned by entmax attention—a sparse alternative to (dense) softmax attention (Peters et al., 2019).

sent8: Finally, sparsity can also be induced via modularity, e.g., by encapsulating task-specific parameters (Ponti et al., 2022).

Output:

How do models leverage sparsity for efficiency, such as MoE method? sent1

1.1. What are some relevant works that follow the MoE concept? sent2

1.1.1 How have recent works scaled transformers using MoE? sent3

1.2. What is the overall performance and resource consumption of MoE models? sent4

1.2.1. Can you provide specific examples of resource consumption reduction in MoE models? sent5

1.2.2. What are some challenges associated with MoE models? sent6

1.3. What is another promising direction for exploiting sparse modeling? sent7

1.4. How can sparsity be induced via modularity? sent8

Content

Paper Title: **<PAPER_TITLE>

**Section Title: **<SECTION_TITLE>

**Content:

**<SECTION_SENTENCE_PREFIXED>

**Output:

C.3 Query generation prompt: generate initial query

System: In this task, you will be given a section from an academic paper, split up into individual sentences.

Also a set of questions based on a partial section in a hierarchical manner, where each question is clearly answered by the sentence indicated by the index.

You need to generate a question that within the scope of the hierarchical question set, requires understanding the corresponding part of this section.

Ideally, this question is based on the sentence that with sufficent following sentences, indicated in the hierarchical question set.

Also provide the suppprintg sentence indexs.

This question needs to be:

Unambiguous: Clearly framed so it does not require follow-up questions for clarification.

1.1 It should be understandable to any expert without needing specific context or jargon found in the given section.

Natural: Phrased as if it is asked by a domain expert conducting research.
Answerable: Should be entirely answerable based on the provided section.
Precise: Question should express a clear information need and not be vague.

4.1 Multiple experts should answer such a question in a similar way.

4.2 The question may need to mention the sub-area and domain specified in the section (especially titles) to be precise.

Requires a long-form, comprehensive answer: not simply extractive or yes-no questions.
The question should not specifically about a particular paper e.g. Artetxe and Schwenk (2019).
The question should be based on the sentence that with sufficent following sentences, indicated in the hierarchical question set.
Less than 20 words.

Content:

Paper Title: Societal Biases in Language Generation: Progress and Challenges

Section Title: Bias Definitions and Metrics

Section content: