Scorio.jl: A Julia package for ranking stochastic responses

TL;DR

Scorio.jl is a Julia package that offers a unified framework for evaluating and ranking systems using various methods, enabling comprehensive analysis of shared task responses.

Contribution

It introduces a versatile Julia package that integrates multiple ranking methods with a common interface for systematic evaluation.

Findings

Effective in synthetic rank recovery tasks

Stable performance under limited trials

Scales well with larger datasets

Abstract

Scorio.jl is a Julia package for evaluating and ranking systems from repeated responses to shared tasks. It provides a common tensor-based interface for direct score-based, pairwise, psychometric, voting, graph, and listwise methods, so the same benchmark can be analyzed under multiple ranking assumptions. We describe the package design, position it relative to existing Julia tools, and report pilot experiments on synthetic rank recovery, stability under limited trials, and runtime scaling.