DEP: A Decentralized Large Language Model Evaluation Protocol

TL;DR

DEP introduces a decentralized, standardized evaluation protocol for large language models that enhances reproducibility, data security, and modularity, enabling long-term, cost-effective benchmarking across diverse tasks.

Contribution

This work presents DEP, a novel decentralized evaluation framework and toolkit that unifies LLM benchmarking, ensuring consistency, data privacy, and ease of adaptation for multiple benchmarks.

Findings

DEP reduces evaluation deployment costs.

Over 60 benchmarks adapted using DEP.

Effective in ensuring secure, reproducible evaluations.

Abstract

With the rapid development of Large Language Models (LLMs), a large number of benchmarks have been proposed. However, most benchmarks lack unified evaluation standard and require the manual implementation of custom scripts, making results hard to ensure consistency and reproducibility. Furthermore, mainstream evaluation frameworks are centralized, with datasets and answers, which increases the risk of benchmark leakage. To address these issues, we propose a Decentralized Evaluation Protocol (DEP), a decentralized yet unified and standardized evaluation framework through a matching server without constraining benchmarks. The server can be mounted locally or deployed remotely, and once adapted, it can be reused over the long term. By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation: benchmark files and evaluation logic stay exclusively on the server side. In remote setting, users cannot access the ground truth, thereby achieving data isolation and leak-proof evaluation. To facilitate practical adoption, we develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control. We also provide detailed documentation for adapting new benchmarks to DEP. Using DEP toolkit, we evaluate multiple LLMs across benchmarks. Experimental results verify the effectiveness of DEP and show that it reduces the cost of deploying benchmark evaluations. As of February 2026, we have adapted over 60 benchmarks and continue to promote community co-construction to support unified evaluation across various tasks and domains.