MINOS: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Text

TL;DR

This paper introduces MINOS, a multimodal evaluation model trained on a high-quality dataset, achieving state-of-the-art performance in evaluating image-text generation tasks across diverse datasets.

Contribution

The paper presents a comprehensive, quality-controlled evaluation dataset and a new evaluation model that outperforms existing open-source models with less training data.

Findings

MINOS achieves state-of-the-art performance across 16 out-of-domain datasets.

Quality control and joint training on I2T and T2I data are crucial for evaluation accuracy.

The model remains competitive with closed-source evaluation models.

Abstract

Evaluation is important for multimodal generation tasks, while traditional multimodal evaluation metrics suffer from several limitations. With the rapid progress of MLLMs, there is growing interest in applying MLLMs to build general evaluation systems. However, existing researches often simply collect large-scale evaluation data for training, while overlooking the quality of evaluation data. What's more, current proposed evaluation models often struggle to achieve consistently strong performance across both image-to-text (I2T) and text-to-image (T2I) tasks. In this paper, through rigorous quality control strategies, we construct a comprehensive multimodal evaluation dataset, Minos-57K, with evaluation samples across 15 datasets, for developing the multimodal evaluation model Minos with SFT and preference alignment training strategies. Notably, despite using less than half the scale of the training data of prior work, our model achieves state-of-the-art evaluation performance across 16 out-of-domain datasets covering both I2T and T2I tasks among all open-source multimodal evaluation models and remain competitive with closed-source models. Extensive experiments demonstrate the importance of leveraging quality control process, jointly training on evaluation data from both I2T and T2I generation tasks and further preference alignment.