Towards Automatic Evaluation for Image Transcreation

TL;DR

This paper introduces a set of automatic evaluation metrics for image transcreation, inspired by machine translation metrics, to assess cultural relevance, semantic equivalence, and visual similarity, aligning well with human judgments.

Contribution

It proposes a comprehensive suite of automatic metrics for image transcreation evaluation, filling the gap of reliance on human assessment and grounded in translation theory and practice.

Findings

VLMs effectively identify cultural relevance and semantic equivalence.

Vision-encoder representations excel at measuring visual similarity.

Metrics show strong correlation (0.55-0.87) with human ratings across 7 countries.

Abstract

Beyond conventional paradigms of translating speech and text, recently, there has been interest in automated transcreation of images to facilitate localization of visual content across different cultures. Attempts to define this as a formal Machine Learning (ML) problem have been impeded by the lack of automatic evaluation mechanisms, with previous work relying solely on human evaluation. In this paper, we seek to close this gap by proposing a suite of automatic evaluation metrics inspired by machine translation (MT) metrics, categorized into: a) Object-based, b) Embedding-based, and c) VLM-based. Drawing on theories from translation studies and real-world transcreation practices, we identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity, and design our metrics to evaluate systems along these axes. Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity. Meta-evaluation across 7 countries shows our metrics agree strongly with human ratings, with average segment-level correlations ranging from 0.55-0.87. Finally, through a discussion of the merits and demerits of each metric, we offer a robust framework for automated image transcreation evaluation, grounded in both theoretical foundations and practical application. Our code can be found here: https://github.com/simran-khanuja/automatic-eval-img-transcreation.