The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning

TL;DR

This paper introduces MosAIC, a multi-agent framework that leverages culturally diverse multimodal models to improve cross-cultural image captioning, addressing Western-centric biases and enhancing cultural representation in AI-generated descriptions.

Contribution

The paper presents a novel multi-agent approach with cultural personas, a new dataset of culturally enriched captions, and a culture-aware evaluation metric for cross-cultural image captioning.

Findings

Multi-agent interaction outperforms single-agent models in cultural captioning.

The dataset includes images from China, India, and Romania with enriched captions.

The culture-adaptable metric effectively evaluates cultural information in captions.

Abstract

Large Multimodal Models (LMMs) exhibit impressive performance across various multimodal tasks. However, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of most data and models. Conversely, multi-agent models have shown significant capability in solving complex tasks. Our study evaluates the collective performance of LMMs in a multi-agent interaction setting for the novel task of cultural image captioning. Our contributions are as follows: (1) We introduce MosAIC, a Multi-Agent framework to enhance cross-cultural Image Captioning using LMMs with distinct cultural personas; (2) We provide a dataset of culturally enriched image captions in English for images from China, India, and Romania across three datasets: GeoDE, GD-VCR, CVQA; (3) We propose a culture-adaptable metric for evaluating cultural information within image captions; and (4) We show that the multi-agent interaction outperforms single-agent models across different metrics, and offer valuable insights for future research. Our dataset and models can be accessed at https://github.com/MichiganNLP/MosAIC.