ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows

TL;DR

This paper introduces ResearchGPT, a benchmark and training dataset for LLMs to assist in end-to-end computer science research workflows, demonstrating that domain-specific high-quality data significantly improves research assistance capabilities.

Contribution

The paper presents CS-54k, a high-quality scientific Q&A corpus, and demonstrates how training on this data enhances LLMs' ability to support scientific research tasks.

Findings

Models trained on CS-50k outperform larger proprietary models.

High-quality domain data improves research assistance capabilities.

CS-4k effectively stratifies LLMs by capability tiers.

Abstract

As large language models (LLMs) advance, the ultimate vision for their role in science is emerging: we could build an AI collaborator to effectively assist human beings throughout the entire scientific research process. We refer to this envisioned system as ResearchGPT. Given that scientific research progresses through multiple interdependent phases, achieving this vision requires rigorous benchmarks that evaluate the end-to-end workflow rather than isolated sub-tasks. To this end, we contribute CS-54k, a high-quality corpus of scientific Q&A pairs in computer science, built from 14k CC-licensed papers. It is constructed through a scalable, paper-grounded pipeline that combines retrieval-augmented generation (RAG) with multi-stage quality control to ensure factual grounding. From this unified corpus, we derive two complementary subsets: CS-4k, a carefully curated benchmark for evaluating AI's ability to assist scientific research, and CS-50k, a large-scale training dataset. Extensive experiments demonstrate that CS-4k stratifies state-of-the-art LLMs into distinct capability tiers. Open models trained on CS-50k with supervised training and reinforcement learning demonstrate substantial improvements. Even 7B-scale models, when properly trained, outperform many larger proprietary systems, such as GPT-4.1, GPT-4o, and Gemini 2.5 Pro. This indicates that making AI models better research assistants relies more on domain-aligned training with high-quality data than on pretraining scale or general benchmark performance. We release CS-4k and CS-50k in the hope of fostering AI systems as reliable collaborators in CS research.