Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset

TL;DR

This study evaluates Transformer-based vision and language models for Brazilian Portuguese image captioning using native and translated datasets, highlighting model performance, biases, and the impact of translation on model generalization.

Contribution

It introduces a cross-native-translated dataset evaluation for Brazilian Portuguese image captioning and compares multiple models, including a new pre-trained VLM, revealing insights into translation effects and biases.

Findings

Swin-DistilBERTimbau outperforms other models in cross-dataset generalization.

ViTucano surpasses larger multilingual models in traditional metrics.

GPT-4 achieves the highest CLIP-Score, indicating strong image-text alignment.

Abstract

Image captioning (IC) refers to the automatic generation of natural language descriptions for images, with applications ranging from social media content generation to assisting individuals with visual impairments. While most research has been focused on English-based models, low-resource languages such as Brazilian Portuguese face significant challenges due to the lack of specialized datasets and models. Several studies create datasets by automatically translating existing ones to mitigate resource scarcity. This work addresses this gap by proposing a cross-native-translated evaluation of Transformer-based vision and language models for Brazilian Portuguese IC. We use a version of Flickr30K comprised of captions manually created by native Brazilian Portuguese speakers and compare it to a version with captions automatically translated from English to Portuguese. The experiments include a cross-context approach, where models trained on one dataset are tested on the other to assess the translation impact. Additionally, we incorporate attention maps for model inference interpretation and use the CLIP-Score metric to evaluate the image-description alignment. Our findings show that Swin-DistilBERTimbau consistently outperforms other models, demonstrating strong generalization across datasets. ViTucano, a Brazilian Portuguese pre-trained VLM, surpasses larger multilingual models (GPT-4o, LLaMa 3.2 Vision) in traditional text-based evaluation metrics, while GPT-4 models achieve the highest CLIP-Score, highlighting improved image-text alignment. Attention analysis reveals systematic biases, including gender misclassification, object enumeration errors, and spatial inconsistencies. The datasets and the models generated and analyzed during the current study are available in: https://github.com/laicsiifes/transformer-caption-ptbr.