SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

TL;DR

SciCustom is a scalable framework that constructs customizable, knowledge-based benchmarks from scientific data to evaluate large language models' specific scientific capabilities.

Contribution

It introduces a novel ontology-grounded knowledge organization and retrieval method for creating fine-grained, application-specific scientific benchmarks without expert annotation.

Findings

Reveals fine-grained differences in LLM capabilities in chemistry and healthcare.

Does not require expert annotation or synthetic question generation.

Enables relevance-aware benchmark retrieval and efficient evaluation.

Abstract

Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabilities required in practice. Most benchmarks are manually curated or domain-generic, limiting scalability and alignment with real scientific use cases. In this paper, we propose a new framework named SciCustom to address the problem. It enables the custom construction of benchmarks from large-scale scientific data to evaluate application-specific scientific capabilities in LLMs. SciCustom first organizes scientific knowledge into ontology-grounded knowledge units with controlled granularity and trains a tagger to map large-scale data instances into this knowledge space. Given a custom requirement, relevant knowledge units are identified via voting-based multi-model consensus. These units enable relevance-aware benchmark retrieval via binary search, followed by proxy subset selection and data-grounded benchmark generation for efficient evaluation. Experiments in chemistry and healthcare demonstrate that SciCustom reveals fine-grained differences in LLM scientific capabilities that standard benchmarks overlook, while requiring neither expert annotation nor synthetic question generation. This work provides a scalable and application-aware foundation for benchmarking scientific capabilities in LLMs. The source code is available at https://github.com/yjwtheonly/SciCustom.