Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics

TL;DR

This paper introduces UniSim-Bench, a comprehensive benchmark for multi-modal perceptual similarity, and explores fine-tuning vision-language models to develop a unified, generalizable perceptual metric that better aligns with human perception.

Contribution

The paper presents UniSim-Bench, a new benchmark for multi-modal perceptual similarity, and demonstrates that fine-tuning models on multiple tasks improves average performance but still faces challenges in generalization.

Findings

General-purpose models perform reasonably well but lag on specific tasks.

Task-specific models do not generalize well to unseen tasks.

Fine-tuned models achieve higher average performance and sometimes surpass task-specific models.

Abstract

Human perception of similarity across uni- and multimodal inputs is highly complex, making it challenging to develop automated metrics that accurately mimic it. General purpose vision-language models, such as CLIP and large multi-modal models (LMMs), can be applied as zero-shot perceptual metrics, and several recent works have developed models specialized in narrow perceptual tasks. However, the extent to which existing perceptual metrics align with human perception remains unclear. To investigate this question, we introduce UniSim-Bench, a benchmark encompassing 7 multi-modal perceptual similarity tasks, with a total of 25 datasets. Our evaluation reveals that while general-purpose models perform reasonably well on average, they often lag behind specialized models on individual tasks. Conversely, metrics fine-tuned for specific tasks fail to generalize well to unseen, though related, tasks. As a first step towards a unified multi-task perceptual similarity metric, we fine-tune both encoder-based and generative vision-language models on a subset of the UniSim-Bench tasks. This approach yields the highest average performance, and in some cases, even surpasses taskspecific models. Nevertheless, these models still struggle with generalization to unseen tasks, highlighting the ongoing challenge of learning a robust, unified perceptual similarity metric capable of capturing the human notion of similarity. The code and models are available at https://github.com/SaraGhazanfari/UniSim.