SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text

TL;DR

SAGEval introduces a novel reference-free evaluation framework for natural language generation that employs a critiquing agent to improve scoring accuracy without relying on ground-truth labels, especially in complex NLG tasks.

Contribution

The paper presents SAGEval, a new framework that enhances LLM-based NLG evaluation by using a critiquing agent to correct scores without needing reference data.

Findings

Critiquing agent effectively rectifies LLM evaluator scores.

SAGEval reduces dependence on labeled data for complex NLG tasks.

Improves evaluation accuracy in reference-free NLG scenarios.

Abstract

Large Language Model (LLM) integrations into applications like Microsoft365 suite and Google Workspace for creating/processing documents, emails, presentations, etc. has led to considerable enhancements in productivity and time savings. But as these integrations become more more complex, it is paramount to ensure that the quality of output from the LLM-integrated applications are relevant and appropriate for use. Identifying the need to develop robust evaluation approaches for natural language generation, wherein references/ground labels doesn't exist or isn't amply available, this paper introduces a novel framework called "SAGEval" which utilizes a critiquing Agent to provide feedback on scores generated by LLM evaluators. We show that the critiquing Agent is able to rectify scores from LLM evaluators, in absence of references/ground-truth labels, thereby reducing the need for labeled data even for complex NLG evaluation scenarios, like the generation of JSON-structured forms/surveys with responses in different styles like multiple choice, likert ratings, single choice questions, etc.