Augmenting Research Ideation with Data: An Empirical Investigation in Social Science
Xiao Liu, Xinyi Dong, Xinyang Gao, Yansong Feng, Xun Pang
TL;DR
This study explores how augmenting large language models with relevant data during research idea generation enhances idea feasibility and quality, especially in social science, through metadata integration and automated validation, leading to more useful and higher-quality ideas.
Contribution
The paper introduces a novel framework that combines data augmentation and automated validation to improve LLM-generated research ideas, demonstrating its effectiveness in social science.
Findings
Metadata increases idea feasibility by 20%.
Automated validation improves idea quality by 7%.
Researchers find augmented ideas highly useful and of higher quality.
Abstract
Recent advancements in large language models (LLMs) demonstrate strong potential for generating novel research ideas, yet such ideas often struggle with feasibility and effectiveness. In this paper, we investigate whether augmenting LLMs with relevant data during the ideation process can improve idea quality. Our framework integrates data at two stages: (1) incorporating metadata during idea generation to guide models toward more feasible concepts, and (2) introducing an automated preliminary validation step during idea selection to assess the empirical plausibility of hypotheses within ideas. We evaluate our approach in the social science domain, with a specific focus on climate negotiation topics. Expert evaluation shows that metadata improves the feasibility of generated ideas by 20%, while automated validation improves the overall quality of selected ideas by 7%. Beyond assessing…
Peer Reviews
Decision·Submitted to ICLR 2026
1. The paper provides a novel, two-pronged framework that effectively integrates empirical data signals (metadata and preliminary validation) directly into the LLM ideation pipeline, which is a significant step beyond existing literature-based approaches. 2. The results are robust, supported by both automatic evaluations using multiple LLM judges and a controlled human expert evaluation that confirms substantial gains in feasibility (20%) and expected effectiveness (18%). 3. The human study succ
1. The investigation is strictly confined to one niche domain (quantitative social science on climate negotiations) using a custom-built dataset (CLIMATEDATABANK). The framework’s transferability to other scientific disciplines or fields with less structured data remains unproven. 2. The data-aware generation process appears to impose an implicit constraint on creativity, resulting in a reported decline in the novelty metric for some experimental settings. This suggests a need to better manage t
- The authors did quite extensive human expert evaluation for all experiments in the paper, which makes the conclusions more convincing. - The empirical results are positive, and human evaluators find the generated ideas helpful and inspiring. - The proposed ideas (metadata conditioning and automatic validation) are extremely simple and easy to implement.
- My biggest concern is that all experiments are done on the social science domain (10 climate negotiation-related research topics in Appendix A), and I'm not sure whether the conclusions could be generalizable across other domains? For example, how would this work for empirical AI research? What would the automatic valiation look like in that case? My biased view is that the automatic validation is only possible for research problems where the execution is quite simple and straightforward, and
1. The paper is clearly written, logically organized, and easy to follow. 2. Applying LLM-based ideation to the social science domain is an interesting and relatively unexplored area. 3. The inclusion of a human study is commendable, as it goes beyond evaluating the LLM’s direct outputs and provides preliminary empirical evidence of the framework’s practical utility. 4. The creation of the CLIMATEDATABANK is a useful resource for future research in this specific social science domain.
1. The proposed data-augmented ideation is a relatively straightforward extension of existing frameworks. Adding dataset descriptions as metadata and performing simple validation are natural incremental steps, not a fundamentally new approach or theoretical contribution. 2. The automatic validation process is largely descriptive and based on keyword counts or correlations, not rigorous statistical or causal analysis. It is unclear how reliable or generalizable these validations are, and they do
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBig Data and Business Intelligence