AutoViVQA: A Large-Scale Automatically Constructed Dataset for Vietnamese Visual Question Answering

TL;DR

This paper introduces AutoViVQA, a large-scale Vietnamese VQA dataset, and explores transformer-based models for Vietnamese visual question answering, comparing evaluation metrics and emphasizing multimodal understanding.

Contribution

It presents AutoViVQA, a new extensive dataset for Vietnamese VQA, and systematically evaluates transformer-based models and metrics in this low-resource language context.

Findings

Transformer models improve Vietnamese VQA performance.

Automatic metrics vary in correlation with human judgment.

Pre-trained multilingual models enhance multimodal understanding.

Abstract

Visual Question Answering (VQA) is a fundamental multimodal task that requires models to jointly understand visual and textual information. Early VQA systems relied heavily on language biases, motivating subsequent work to emphasize visual grounding and balanced datasets. With the success of large-scale pre-trained transformers for both text and vision domains -- such as PhoBERT for Vietnamese language understanding and Vision Transformers (ViT) for image representation learning -- multimodal fusion has achieved remarkable progress. For Vietnamese VQA, several datasets have been introduced to promote research in low-resource multimodal learning, including ViVQA, OpenViVQA, and the recently proposed ViTextVQA. These resources enable benchmarking of models that integrate linguistic and visual features in the Vietnamese context. Evaluation of VQA systems often employs automatic metrics originally designed for image captioning or machine translation, such as BLEU, METEOR, CIDEr, Recall, Precision, and F1-score. However, recent research suggests that large language models can further improve the alignment between automatic evaluation and human judgment in VQA tasks. In this work, we explore Vietnamese Visual Question Answering using transformer-based architectures, leveraging both textual and visual pre-training while systematically comparing automatic evaluation metrics under multilingual settings.