SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence

TL;DR

SciEvalKit is an open-source toolkit designed to evaluate AI models' scientific capabilities across multiple disciplines and tasks, promoting standardized, reproducible benchmarking for scientific intelligence.

Contribution

It introduces a comprehensive, domain-diverse evaluation platform tailored for scientific AI models, supporting multiple scientific tasks and datasets with flexible, reproducible benchmarking features.

Findings

Supports six major scientific domains.

Provides expert-grade, real-world datasets.

Enables transparent, reproducible evaluations.

Abstract

We introduce SciEvalKit, a unified benchmarking toolkit designed to evaluate AI models for science across a broad range of scientific disciplines and task capabilities. Unlike general-purpose evaluation platforms, SciEvalKit focuses on the core competencies of scientific intelligence, including Scientific Multimodal Perception, Scientific Multimodal Reasoning, Scientific Multimodal Understanding, Scientific Symbolic Reasoning, Scientific Code Generation, Science Hypothesis Generation and Scientific Knowledge Understanding. It supports six major scientific domains, spanning from physics and chemistry to astronomy and materials science. SciEvalKit builds a foundation of expert-grade scientific benchmarks, curated from real-world, domain-specific datasets, ensuring that tasks reflect authentic scientific challenges. The toolkit features a flexible, extensible evaluation pipeline that enables batch evaluation across models and datasets, supports custom model and dataset integration, and provides transparent, reproducible, and comparable results. By bridging capability-based evaluation and disciplinary diversity, SciEvalKit offers a standardized yet customizable infrastructure to benchmark the next generation of scientific foundation models and intelligent agents. The toolkit is open-sourced and actively maintained to foster community-driven development and progress in AI4Science.