Evaluating the Evaluation of Diversity in Commonsense Generation

TL;DR

This paper systematically evaluates various diversity metrics for commonsense generation, revealing that content-based metrics align better with human judgments and are more reliable than form-based metrics, which tend to overestimate diversity.

Contribution

It provides a comprehensive meta-evaluation of diversity metrics, introduces an LLM-annotated dataset, and recommends content-based metrics for future evaluations.

Findings

Form-based metrics overestimate diversity.

Content-based metrics correlate better with LLM ratings.

Content metrics are more reliable for diversity evaluation.

Abstract

In commonsense generation, given a set of input concepts, a model must generate a response that is not only commonsense bearing, but also capturing multiple diverse viewpoints. Numerous evaluation metrics based on form- and content-level overlap have been proposed in prior work for evaluating the diversity of a commonsense generation model. However, it remains unclear as to which metrics are best suited for evaluating the diversity in commonsense generation. To address this gap, we conduct a systematic meta-evaluation of diversity metrics for commonsense generation. We find that form-based diversity metrics tend to consistently overestimate the diversity in sentence sets, where even randomly generated sentences are assigned overly high diversity scores. We then use an Large Language Model (LLM) to create a novel dataset annotated for the diversity of sentences generated for a commonsense generation task, and use it to conduct a meta-evaluation of the existing diversity evaluation metrics. Our experimental results show that content-based diversity evaluation metrics consistently outperform the form-based counterparts, showing high correlations with the LLM-based ratings. We recommend that future work on commonsense generation should use content-based metrics for evaluating the diversity of their outputs.