BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answering

TL;DR

This paper introduces BARTPhoBEiT, a transformer-based model for Vietnamese Visual Question Answering that combines sequence-to-sequence and image transformers, achieving state-of-the-art results on Vietnamese VQA datasets.

Contribution

The paper presents the first Vietnamese-specific transformer model for VQA, integrating pre-trained sequence-to-sequence and image transformers to improve performance.

Findings

Outperforms baseline models on Vietnamese VQA datasets

Achieves state-of-the-art results in six evaluation metrics

Demonstrates effectiveness of combined language and image transformers in Vietnamese

Abstract

Visual Question Answering (VQA) is an intricate and demanding task that integrates natural language processing (NLP) and computer vision (CV), capturing the interest of researchers. The English language, renowned for its wealth of resources, has witnessed notable advancements in both datasets and models designed for VQA. However, there is a lack of models that target specific countries such as Vietnam. To address this limitation, we introduce a transformer-based Vietnamese model named BARTPhoBEiT. This model includes pre-trained Sequence-to-Sequence and bidirectional encoder representation from Image Transformers in Vietnamese and evaluates Vietnamese VQA datasets. Experimental results demonstrate that our proposed model outperforms the strong baseline and improves the state-of-the-art in six metrics: Accuracy, Precision, Recall, F1-score, WUPS 0.0, and WUPS 0.9.