Fusion-Eval: Integrating Assistant Evaluators with LLMs

TL;DR

Fusion-Eval is a novel method that uses large language models to combine insights from multiple specialized assistant evaluators, significantly improving correlation with human judgments in natural language system evaluation.

Contribution

It introduces a new approach that leverages LLMs to integrate diverse evaluator scores, enhancing the accuracy of automatic evaluation metrics for language systems.

Findings

Achieves 0.962 system-level Kendall-Tau correlation with humans on SummEval.

Attains 0.744 turn-level Spearman correlation on TopicalChat.

Outperforms baseline evaluation methods significantly.

Abstract

Evaluating natural language systems poses significant challenges, particularly in the realms of natural language understanding and high-level reasoning. In this paper, we introduce 'Fusion-Eval', an innovative approach that leverages Large Language Models (LLMs) to integrate insights from various assistant evaluators. The LLM is given the example to evaluate along with scores from the assistant evaluators. Each of these evaluators specializes in assessing distinct aspects of responses. Fusion-Eval achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.744 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. These results highlight Fusion-Eval's significant potential in the realm of natural language system evaluation.