A Transparent Fairness Evaluation Protocol for Open-Source Language Model Benchmarking on the Blockchain

TL;DR

This paper presents a transparent, blockchain-based protocol for fair benchmarking of open-source language models, ensuring verifiability and reproducibility while evaluating bias and fairness across multiple datasets and languages.

Contribution

It introduces a novel onchain evaluation protocol using smart contracts for fair benchmarking of LLMs, with open-source code and multi-language fairness analysis.

Findings

Benchmarking Llama, DeepSeek, and Mistral models on PISA dataset.

Evaluation of social bias using Context Association Metrics.

Cross-linguistic disparities observed in multilingual fairness assessments.

Abstract

Large language models (LLMs) are increasingly deployed in realworld applications, yet concerns about their fairness persist especially in highstakes domains like criminal justice, education, healthcare, and finance. This paper introduces transparent evaluation protocol for benchmarking the fairness of opensource LLMs using smart contracts on the Internet Computer Protocol (ICP) blockchain (Foundation, 2023). Our method ensures verifiable, immutable, and reproducible evaluations by executing onchain HTTP requests to hosted Hugging Face endpoints and storing datasets, prompts, and metrics directly onchain. We benchmark the Llama, DeepSeek, and Mistral models on the PISA dataset for academic performance prediction (OECD, 2018), a dataset suitable for fairness evaluation using statistical parity and equal opportunity metrics (Hardt et al., 2016). We also evaluate structured Context Association Metrics derived from the StereoSet dataset (Nadeem et al., 2020) to measure social bias in contextual associations. We further extend our analysis with a multilingual evaluation across English, Spanish, and Portuguese using the Kaleidoscope benchmark (Salazar et al., 2025), revealing cross-linguistic disparities. All code and results are open source, enabling community audits and longitudinal fairness tracking across model versions.