MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences

TL;DR

MetaMetrics is a supervised meta-metric that calibrates and combines existing evaluation metrics to better align with human preferences across language and vision generation tasks, improving evaluation accuracy.

Contribution

It introduces MetaMetrics, a novel supervised approach to calibrate and optimize existing metrics for better alignment with human preferences in diverse generation tasks.

Findings

MetaMetrics closely aligns with human preferences in multiple tasks

It improves evaluation accuracy across multilingual and multi-domain scenarios

MetaMetrics is flexible and easily integrable into various applications

Abstract

Understanding the quality of a performance evaluation metric is crucial for ensuring that model outputs align with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics, a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. MetaMetrics aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes MetaMetrics a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts.