GUIDE: Towards Scalable Advising for Research Ideas

TL;DR

This paper presents GUIDE, a scalable advising system that uses a small, well-structured model and literature database to outperform larger models in research hypothesis evaluation, achieving high acceptance rates for proposals.

Contribution

The paper introduces a novel advising framework that combines a small model, literature compression, and structured reasoning to improve research hypothesis assessment.

Findings

Small models with structured reasoning outperform large models in acceptance rates.

High-confidence predictions exceed 90% acceptance on ICLR 2025 test set.

Structured reasoning enhances hypothesis evaluation quality.

Abstract

The field of AI research is advancing at an unprecedented pace, enabling automated hypothesis generation and experimental design across diverse domains such as biology, mathematics, and artificial intelligence. Despite these advancements, there remains a significant gap in the availability of scalable advising systems capable of providing high-quality, well-reasoned feedback to refine proposed hypotheses and experimental designs. To address this challenge, we explore key factors that underlie the development of robust advising systems, including model size, context length, confidence estimation, and structured reasoning processes. Our findings reveal that a relatively small model, when equipped with a well-compressed literature database and a structured reasoning framework, can outperform powerful general-purpose language models such as Deepseek-R1 in terms of acceptance rates for self-ranked top-30% submissions to ICLR 2025. Moreover, when limited to high-confidence predictions, our system achieves an acceptance rate exceeding 90% on the ICLR 2025 test set, underscoring its potential to significantly enhance the quality and efficiency of hypothesis generation and experimental design. The code is released at https://github.com/HowardLiu0830/GUIDE-Research-Idea-Evaluation.