UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities

TL;DR

UniDial-EvalKit is a comprehensive, standardized toolkit designed to evaluate multi-turn interactive AI systems efficiently, reproducibly, and extensibly across diverse benchmarks, facilitating fair comparison and accelerating progress.

Contribution

It introduces a unified, modular evaluation framework that standardizes data formats, streamlines pipelines, and supports large-scale, reproducible assessments of conversational AI.

Findings

Standardizes heterogeneous data formats into a universal schema

Streamlines evaluation pipelines with modular architecture

Enhances evaluation efficiency and reproducibility

Abstract

Benchmarking AI systems in multi-turn interactive scenarios is essential for understanding their practical capabilities in real-world applications. However, existing evaluation protocols are highly heterogeneous, differing significantly in dataset formats, model interfaces, and evaluation pipelines, which severely impedes systematic comparison. In this work, we present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems. The core contribution of UDE lies in its holistic unification: it standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface. It also supports efficient large-scale evaluation through parallel generation and scoring, as well as checkpoint-based caching to eliminate redundant computation. Validated across diverse multi-turn benchmarks, UDE not only guarantees high reproducibility through standardized workflows and transparent logging, but also significantly improves evaluation efficiency and extensibility. We make the complete toolkit and evaluation scripts publicly available to foster a standardized benchmarking ecosystem and accelerate future breakthroughs in interactive AI.