Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

TL;DR

This paper defines Scientific General Intelligence (SGI) for AI, introduces a benchmark with tasks inspired by scientific inquiry, evaluates LLMs revealing significant gaps, and proposes TTRL to improve hypothesis novelty during inference.

Contribution

It provides a formal SGI definition based on the Practical Inquiry Model, creates a comprehensive benchmark for scientific reasoning, and introduces TTRL to enhance LLMs' scientific hypothesis generation.

Findings

Low exact match (10-20%) in deep research tasks

Ideas often lack feasibility and detail

Dry experiments have high code executability but low accuracy

Abstract

Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.