SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks

TL;DR

SciArena is a community-driven platform for evaluating scientific literature-grounded models, leveraging collective human judgments to rank model performance and promote research in automated evaluation methods.

Contribution

It introduces SciArena, an open platform for collaborative evaluation of scientific models, and releases SciArena-Eval, a benchmark for automated model assessment based on human preferences.

Findings

High-quality data collected from 20,000+ votes

Model rankings reveal insights into model performance

Benchmark highlights challenges in automated evaluation

Abstract

We present SciArena, an open and collaborative platform for evaluating foundation models on scientific literature-grounded tasks. Unlike traditional benchmarks for scientific literature understanding and synthesis, SciArena engages the research community directly, following the Chatbot Arena evaluation approach of community voting on model comparisons. By leveraging collective intelligence, SciArena offers a community-driven evaluation of model performance on open-ended scientific tasks that demand literature-grounded, long-form responses. The platform currently supports 47 foundation models and has collected over 20,000 votes from human researchers across diverse scientific domains. Our analysis of the data collected so far confirms its high quality. We discuss the results and insights based on the model ranking leaderboard. To further promote research in building model-based automated evaluation systems for literature tasks, we release SciArena-Eval, a meta-evaluation benchmark based on collected preference data. It measures the accuracy of models in judging answer quality by comparing their pairwise assessments with human votes. Our experiments highlight the benchmark's challenges and emphasize the need for more reliable automated evaluation methods.