Jury: A Comprehensive Evaluation Toolkit

TL;DR

Jury is an open-source toolkit designed to standardize and streamline the evaluation process across various NLP tasks and metrics, addressing the fragmentation in current evaluation practices.

Contribution

It introduces a unified framework for evaluation in NLP, facilitating consistent and comprehensive assessment across diverse tasks and metrics.

Findings

Widespread adoption of jury since release

Improved consistency in NLP system evaluations

Facilitated comparison across different NLP models

Abstract

Evaluation plays a critical role in deep learning as a fundamental block of any prediction-based system. However, the vast number of Natural Language Processing (NLP) tasks and the development of various metrics have led to challenges in evaluating different systems with different metrics. To address these challenges, we introduce jury, a toolkit that provides a unified evaluation framework with standardized structures for performing evaluation across different tasks and metrics. The objective of jury is to standardize and improve metric evaluation for all systems and aid the community in overcoming the challenges in evaluation. Since its open-source release, jury has reached a wide audience and is available at https://github.com/obss/jury.