REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation
Jun Gao, Wei Bi, Ruifeng Xu, Shuming Shi

TL;DR
This paper introduces REAM#, a method to enhance reference-based evaluation metrics for open-domain dialogue systems by predicting and augmenting reference sets to improve their reliability and correlation with human judgments.
Contribution
It proposes a prediction model to assess and improve the quality of reference sets, thereby increasing the reliability of reference-based evaluation metrics.
Findings
The prediction model effectively estimates reference set reliability.
Augmenting reference sets with predicted high-quality references improves metric reliability.
Enhanced metrics show better correlation with human judgments.
Abstract
The lack of reliable automatic evaluation metrics is a major impediment to the development of open-domain dialogue systems. Various reference-based metrics have been proposed to calculate a score between a predicted response and a small set of references. However, these metrics show unsatisfactory correlations with human judgments. For a reference-based metric, its reliability mainly depends on two factors: its ability to measure the similarity between the predicted response and the reference response, as well as the reliability of the given reference set. Yet, there are few discussions on the latter. Our work attempts to fill this vacancy. We first clarify an assumption on reference-based metrics that, if more high-quality references are added into the reference set, the reliability of the metric will increase. Next, we present REAM: an enhancement approach to Reference-based…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Speech and dialogue systems · Natural Language Processing Techniques
