CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants

TL;DR

This paper introduces practical, repeatable benchmarks tailored for evaluating LLMs in high-energy physics and high-performance computing contexts, focusing on documentation, code generation, and data analysis tasks.

Contribution

The paper develops a comprehensive benchmark suite specifically designed for LLMs in HEP/HPC, emphasizing repeatability, domain relevance, and multi-faceted evaluation metrics.

Findings

Benchmarks enable fair comparison of models in scientific coding tasks.

Automated scoring facilitates consistent evaluation across experiments.

The suite highlights failure modes specific to HEP/HPC software development.

Abstract

Large Language Models (LLM) are increasingly used for software development, yet existing benchmarks for LLM-based coding assistance do not reflect the constraints of High Energy Physics (HEP) and High Performance Computing (HPC) software. Code correctness must respect science constraints and changes must integrate into large, performance-critical codebases with complex dependencies and build systems. The primary contribution of this paper is the development of practical, repeatable benchmarks that quantify LLM performance on HEP/HPC-relevant tasks. We introduce three evaluation tracks -- code documentation benchmarks measure the ability of an LLM to generate Doxygen-style comments, code generation benchmarks evaluate end-to-end usability on representative GPU kernels, and graphical data analysis benchmarks evaluate vision-enabled LLMs. These benchmarks provide a unified framework for measuring progress in scientific coding assistance across documentation quality, code generation robustness, and multimodal validation analysis. By emphasizing repeatability, automated scoring, and domain-relevant failure modes, the suite enables fair comparisons of models and settings while supporting future work on methods that improve reliability for HEP/HPC software development.