EVJVQA Challenge: Multilingual Visual Question Answering

TL;DR

The paper introduces EVJVQA, a multilingual VQA benchmark dataset in Vietnamese, English, and Japanese, designed to evaluate and motivate research on culturally specific visual question answering systems.

Contribution

It provides a new multilingual dataset with over 33,000 question-answer pairs on Vietnamese images and reports on a challenge with top systems using ViT and mT5 models.

Findings

Top F1-score of 0.4392 and BLEU of 0.4009 on private test set.

Multilingual VQA systems benefit from pre-trained transformer models.

The dataset and challenge foster further research in culturally aware multilingual VQA.

Abstract

Visual Question Answering (VQA) is a challenging task of natural language processing (NLP) and computer vision (CV), attracting significant attention from researchers. English is a resource-rich language that has witnessed various developments in datasets and models for visual question answering. Visual question answering in other languages also would be developed for resources and models. In addition, there is no multilingual dataset targeting the visual content of a particular country with its own objects and cultural characteristics. To address the weakness, we provide the research community with a benchmark dataset named EVJVQA, including 33,000+ pairs of question-answer over three languages: Vietnamese, English, and Japanese, on approximately 5,000 images taken from Vietnam for evaluating multilingual VQA systems or models. EVJVQA is used as a benchmark dataset for the challenge of multilingual visual question answering at the 9th Workshop on Vietnamese Language and Speech Processing (VLSP 2022). This task attracted 62 participant teams from various universities and organizations. In this article, we present details of the organization of the challenge, an overview of the methods employed by shared-task participants, and the results. The highest performances are 0.4392 in F1-score and 0.4009 in BLUE on the private test set. The multilingual QA systems proposed by the top 2 teams use ViT for the pre-trained vision model and mT5 for the pre-trained language model, a powerful pre-trained language model based on the transformer architecture. EVJVQA is a challenging dataset that motivates NLP and CV researchers to further explore the multilingual models or systems for visual question answering systems. We released the challenge on the Codalab evaluation system for further research.

Tables

Table 1. Table 1: Schedule of the VLSP 2021 - ViMRC Challenge.

Time	Phase
October 1st	Trial Data
October 5th	Public test
November 10th	Private test
November 12th	Competition end
November 20th	Submission deadline
November 23th	Notification of acceptance
November 25th	Camera-ready due

Table 2. Table 2: Participation summary of the VLSP 2022 - EVJVQA Challenge.

Metric	Value
#Registration Teams	62
#Joined Teams	57
#Signed Data Agreements	36
#Submitted Teams	8
#Paper Submissions	5

Table 3. Table 3: Number of QAs in each language in our UIT-EVJVQA dataset.

	Training	Public test	Private test	Total
Vietnamese	8,334	1,685	1,670	11,689
English	7,189	1,679	1,671	10,539
Japanese	8,262	1,651	1,649	11,562
Total	23,785	5,015	4,990	33,790

Table 4. Table 4: Statistic of question and answer in the UIT-EVJVQA dataset.

	Question			Answer
	Max.	Min.	Avg.	Max.	Min.	Avg.
Vietnamese	22	3	8.7	32	1	7.2
English	26	3	8.6	23	1	5.0
Japanese	45	4	13.3	23	1	5.9

Table 5. Table 5: Final results of submitted methods.

No.	Team name	Model type	Models	Public Test		Private Test
				F1	BLEU	F1	BLEU
1	CIST AI	Single	ViT + mT5	0.3491	0.2508	0.4392	0.4009
2	OhYeah	Single	ViT + mT5	0.5755	0.4866	0.4349	0.3868
3	DS-STBFL	Ensemble	CNN-Seq2Seq + ViT + OFA	0.3390	0.2156	0.4210	0.3482
4	FCoin	Single	ViT + mBERT	0.3355	0.2437	0.4103	0.3549
5	VL-UIT	Single	BEiT + CLIP + Detectron-2 + mBERT + BM25 + FastText	0.3053	0.1878	0.3663	0.2743
6	BDboi	Ensemble	ViT + BEiT + SwinTransformer + CLIP + OFA + BLIP	0.3023	0.2183	0.3164	0.2649
7	UIT-squad	Ensemble	VinVL+mBERT	0.3224	0.2238	0.3024	0.1667
8	VC-Internship	Single	ResNet-152 + OFA	0.3017	0.1639	0.3007	0.1337
9	Baseline	Single	ViT + mBERT	0.2924	0.2183	0.3346	0.2275

Table 6. Table 6: Results of the top-5 methods in English part on the private test set. S stands for short, M stands for medium, L stands for long, and XL stands for very long.

		Top 1		Top 2		Top 3		Top 4		Top 5
		F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU
Question	S	0.3900	0.1867	0.5000	0.1250	0.4261	0.2022	0.4308	0.2184	0.3728	0.1535
	M	0.3935	0.2218	0.4922	0.2961	0.3833	0.1889	0.3730	0.1909	0.3222	0.1329
	L	0.3815	0.2044	0.4470	0.2832	0.3475	0.1628	0.3248	0.1549	0.2982	0.1093
	XL	0.3681	0.2019	0.4368	0.2838	0.3059	0.1518	0.3137	0.1253	0.3030	0.1288
Answer	S	0.3000	0.1308	0.2429	0.0975	0.2999	0.1209	0.3004	0.1282	0.2498	0.0912
	M	0.4751	0.2912	0.3478	0.1994	0.4603	0.2530	0.4502	0.2553	0.4013	0.1737
	L	0.4742	0.2948	0.5063	0.3522	0.4473	0.2237	0.3837	0.1897	0.3642	0.1517
	XL	0.4456	0.1580	0.5709	0.4029	0.3882	0.1049	0.3798	0.1175	0.4219	0.1392

Table 7. Table 7: Results of the top-5 methods in Vietnamese part on the private test set. S stands for short, M stands for medium, L stands for long, and XL stands for very long.

		Top 1		Top 2		Top 3		Top 4		Top 5
		F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU
Question	S	0.4143	0.2088	0.4370	0.2412	0.4582	0.2482	0.4894	0.2768	0.4116	0.1909
	M	0.4997	0.3259	0.4976	0.3150	0.5008	0.3111	0.4879	0.3016	0.4347	0.2271
	L	0.4797	0.2954	0.4582	0.2582	0.4600	0.2687	0.4423	0.2545	0.4014	0.1835
	XL	0.4197	0.2446	0.4349	0.2083	0.4430	0.2359	0.4121	0.2234	0.3154	0.1447
Answer	S	0.3621	0.1659	0.3719	0.1700	0.3692	0.1666	0.3576	0.1656	0.3493	0.1481
	M	0.5564	0.3874	0.5507	0.3743	0.5467	0.3626	0.5433	0.3609	0.4705	0.2604
	L	0.5326	0.3623	0.4859	0.2849	0.5141	0.3244	0.4802	0.2851	0.4007	0.1760
	XL	0.4898	0.3172	0.4627	0.1959	0.5328	0.3253	0.4142	0.2243	0.3430	0.1278

Table 8. Table 8: Results of the top-5 methods in Japanese partition on the private test set in. S stands for short, M stands for medium, L stands for long, and XL stands for very long.

		Top 1		Top 2		Top 3		Top 4		Top 5
		F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU	F1	BLEU
Question	S	0.7500	0.3125	0.4060	0.1917	0.6667	0.1850	0.4444	0.1250	0.3333	0.0460
	M	0.4494	0.2004	0.3905	0.2164	0.4634	0.2984	0.4841	0.3118	0.4485	0.2447
	L	0.4489	0.2841	0.3679	0.1974	0.4395	0.2720	0.4150	0.2741	0.4058	0.2333
	XL	0.4399	0.2915	0.3456	0.1863	0.3896	0.2333	0.3834	0.2453	0.3452	0.1815
Answer	S	0.1983	0.075	0.3101	0.1336	0.2057	0.0713	0.1809	0.0714	0.1983	0.0750
	M	0.3503	0.1942	0.4654	0.2822	0.3236	0.1750	0.3443	0.2095	0.3503	0.1942
	L	0.5242	0.3667	0.4436	0.2731	0.4734	0.3156	0.4609	0.3230	0.5242	0.3667
	XL	0.5911	0.4245	0.3684	0.1586	0.5131	0.3321	0.4983	0.3358	0.5911	0.4245

Table 9. Table 9: F1-score for side predictions of every team for each language

Team	Vietnamese	English	Japanese
CIST AI	0.4948	0.3889	0.3922
OhYeah	0.3814	0.3235	0.4085
DS-STBFL	0.4811	0.3137	0.3366
FCoin	0.4021	0.3039	0.3595
VL-UIT	0.2268	0.2418	0.1471

Table 10. Table 10: F1-score for color predictions of every team for each language

Team	Vietnamese	English	Japanese
CIST AI	0.3230	0.3434	0.3450
OhYeah	0.2954	0.3326	0.3229
DS-STBFL	0.5161	0.4933	0.3934
FCoin	0.3619	0.3142	0.3234
VL-UIT	0.2716	0.2420	0.2355

Equations

$$Precision(P)=\frac{GA\cap PA}{PA}$$

$$Recall(R)=\frac{GA\cap PA}{GA}$$

$$F1=\frac{2PR}{P+R}$$

Full text

1 Introduction

Visual or Image-Based Question Answering is a challenging task that requires knowledge of two hot AI fields: natural language processing and computer vision. Specifically, querying the information of images through human-language questions is a friendly and natural approach to searching for information, meeting the needs of people extracting information in many domains such as life, education, work, etc. However, studies have mainly focused on resource-rich languages such as English. In this challenge, we aim to extend visual question answering to more languages, including rich and low-resource languages, including resources and models.

English has witnessed numerous benchmarks for evaluating and developing visual question answering models or systems. Recently, researchers designed datasets with goal-oriented evaluations. Firstly, VQA datasets Antol et al. (2015); Zhu et al. (2016); Goyal et al. (2017); Changpinyo et al. (2022a) were created on general images. Soon after, the more complex dataset based on reasoning is discovered by Hudson and Manning (2019). In addition, VQA Gurari et al. (2018) is also geared towards support applications for seemingly-impaired and blind people. Singh et al. (2019) showed that VQA has the ability to read text on photos. More challenging, VQA requires external, commonsense, or world knowledge to predict more correct answers Marino et al. (2019); Schwenk et al. (2022). Besides, Changpinyo et al. Changpinyo et al. (2022b) proposed a multilingual dataset for visual question answering in 13 languages. However, this dataset is done automatically based on the auto-translation and verification method. In this paper, we presented a human-generation multilingual dataset, including three languages: English (resource-rich language), Japanese, and Vietnamese(resource-poor language).

In this paper, we have three main contributions described as follows.

Firstly, we constructed UIT-EVJVQA, a multilingual dataset for evaluating the visual question answering systems or models, which comprises 33.790 question-answer pairs in three languages: English, Vietnamese, and Japanese. 2. 2.

We organize the VLSP2022-EVJVQA Challenge for evaluating multilingual VQA models (Vietnamese, English, and Japanese) at the VLSP 2022. Our baseline system obtains 0.3346 in F1-score and 0.2275 in BLEU on the public and private test sets, respectively, and there are no models of participating teams that pass 0.44 (in F1-score) on the private test set, which indicates our dataset is challenging and requires the development of multilingual VQA models. 3. 3.

When combined with other VQA datasets for analysis, UIT-EVJVQA could potentially be a useful resource for multilingual research.

The following is how the rest of the article is organized. In Section 2, we provide a brief overview of the background and relevant studies. We introduce the VLSP 2021-EVJVQA Challenge in Section 3. Our new dataset (UIT-EVJVQA) is presented in detail in Section 4. Section 5 presents the systems and results proposed by participating teams. In Section 6, we provide further analysis of the challenge results. Finally, Section 7 summarizes the findings of the VLSP 2022-EVJVQA Challenge and suggests future research directions.

2 Background and Related Work

Visual Question Answering (VQA) is a challenging task that has significant value not only in the research community but also in daily life. VQA task was first introduced by Antol et al. (2015). The authors were successful in creating a novel dataset and fundamental English methodologies. Inspired by that success, various further studies have been created and implemented in a variety of languages Gupta (2017) including Chinese Qi et al. (2022), Japanese Shimizu et al. (2018), and Vietnamese Tran et al. (2021a).

VQA has gained more attention from researchers in recent years and has shown significant growth. Studies are currently being introduced not only in monolingual but also in multilingual applications Pfeiffer et al. (2021); Khan et al. (2021); Liu et al. (2022); Nooralahzadeh and Sennrich (2022). This stage contributes significantly to the creation of multilingual VQA (mVQA) systems. Some typical research works in this approach can be mentioned such as Gupta et al. (2020) with the study that the proposed model is capable of predicting responses from questions in Hindi, English, or Code-mixed (Hinglish: Hindi-English) languages; Changpinyo et al. Changpinyo et al. (2022b) with a translation-based mVQA dataset in 7 distinct languages; Gao et al. Gao et al. (2015) construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations.

In this study, we create the first dataset for the task of mVQA on English-Vietnamese-Japanese (EVJVQA). The EVJVQA dataset is expected to open up new research areas and aid in evaluating multilingual VQA models.

3 The VLSP 2022 - EVJVQA Challenge

3.1 Task Definition

This task aims to enable the ability of computers to understand images and answer relevant questions in different languages from users. The task is defined as below (Figure 1):

• •

Input: Given an image and a question that can be answerable.

• •

Output: An answer where can be a span related to the image’s content.

Language Selection. Three languages are selected in which the main language is Vietnamese, and the other two popular other languages (English and Japanese) in the pictures taken from Vietnam.

3.2 Evaluation Metrics

In this challenge, we use two evaluation metrics: F1 and BLUE. Based on Rajpurkar et al. (2016), the F1-score of each answer is calculated based on tokens of the gold answer (GA) and tokens of the predicted answer (PA). The overall F1 is averaged across all questions of each set. For Vietnamese and English languages, we calculate F1 based on tokens, whereas F1 is calculated based on characters for Japanese.

[TABLE]

[TABLE]

[TABLE]

Inspired by Papineni et al. (2002), the Bilingual Evaluation Understudy (BLEU), a popular evaluation metric in machine translation, computes the n-gram co-occurrence between human-generation answers and system-generation answers. The best performances were estimated by averaged BLEU-based performances (BLEU-1, BLEU-2, BLEU-3, and BLEU-4) on the public test and private test sets. Both evaluation metrics ignore punctuations.

3.3 Schedule and Overview Summary

Table 1 shows important dates of the VLSP 2022 - EVJVQA Challenge. It lasted for two months, during which the participating teams spent 39 days developing the multilingual visual question answering systems.

Besides, Table 2 presents an overview of the participating teams who joined the VLSP2022-EVJVQA.

4 Corpus Creation

A previous work Tran et al. (2021b) inherited assets from the well-known VQA benchmark in English and the COCO-QA Ren et al. (2015a), then they proposed a semi-automatic annotating system by using machine translation to translate question-answer pairs from English to Vietnamese. On the other hand, we argue that the context in images captured in Vietnam is more complicated than in images coming from VQA benchmarks in English because of its crowded scene and ”out-of-common” objects, or in particular, objects that are not commonly used outside of Vietnam. Moreover, using such a machine translation system as Tran et al. (2021b) is hard to ensure the natural aspect of using language, which caused lots of confusion while evaluating VQA methods in Vietnamese. To overcome the above flaws and research and develop a VQA system, particularly for the Vietnamese, we constructed the novel dataset with images collected manually and relevant to daily life in Vietnam. In addition, to address and challenge the research community, we provided our dataset multilingual question-answer pairs to encourage the research community to explore and propose an effective system that can answer questions written in various languages.

4.1 Image collection

To build a VQA dataset in the Vietnamese context, we search for images with a diverse and open-domain set of keywords. We first selectively prepare various keywords which are relevant to Vietnamese locations and daily life activities or result in images that specifically contain targeted objects in Vietnamese scenes. For instance, the keywords can be Vietnamese streets, markets, sidewalk eateries, cultural sites, human outdoor activities, means of transport, or house interiors. Some keywords are appended with Vietnamese location names like Hanoi or Saigon for more variation in geological and cultural context. We then use these keywords to scrap images from Google Images. The image-scraping process is facilitated by the Octoparse tool.

After collecting the images, we proceed with the filtering stage. The images originally came in various sizes. However, we must ensure the details in them are clearly visible. Therefore, we only keep images with widths and heights greater than 500 and 400, respectively. We also filter out GIF files and other file formats apart from JPEG and PNG. As a result, we obtain 4,909 images with their size varying in the range of 500 - 6000 pixels in terms of width and the range of 400 - 4032 pixels in terms of height.

4.2 Questions and answers creation process

The questions and answers (QAs) of the UIT-EVJVQA dataset are first created in Vietnamese throughout the set of images. Afterward, these QAs are translated into English and Japanese. Both stages are conducted by the source of crowd workers. QAs in all three languages are then merged according to the images and eventually constitute the final corpus. The overall pipeline of the aforementioned process is visualized in Figure 2.

4.2.1 Base Vietnamese QAs creation

We first employ five crowd workers for the Vietnamese questions and answers creation stage. For each image, the workers are asked to formulate 3-5 question-answer pairs based on the details and objects that appear in the visible scene. The workers are required to conform to the following guideline:

• •

Encourage using phrases or full sentences to give the answers.

• •

Restrict the use of single words as answers.

• •

No selective question or yes/no question allowed.

• •

Numbers must be typed in alphabet characters rather than numeric characters, and they must not be greater than 10.

• •

For colors, only use provided colors such as black, white, red, orange, yellow, green, blue, pink, purple, brown, and grey. If these colors cannot exactly describe the true color of the object, ignore such color property in the sentence.

• •

In the case of mentioning direction, if following that direction words is an object, then such direction is defined based on that object, else using the perspective of the annotator to define the direction.

Eventually, this stage yields 11,689 pairs of question and answer in Vietnamese for our dataset.

4.2.2 English and Japanese QAs human-translation

All the Vietnamese QAs are henceforth passed through the human-translation stages. These stages demand the employment of qualified crowd workers for the translation of questions and answers. For English translation, the workers must have at least IELTS certification with an overall band score of 6.5. Meanwhile, translators for the Japanese translation task must achieve an N3 proficiency level or above. Overall, there were seven and nine translators working simultaneously for the English and Japanese translation process, respectively. The English and Japanese translation stages also have their corresponding guidelines.

English translation guideline: The English QAs are translated from the Vietnamese ones, with many entities and attributes in the sentences retained during the translation as possible. The translators are encouraged to use phrases or full sentences as the translated answers. Apostrophes in sentences are restricted. Thus, translators should use the uncontracted form or other valid grammar formulations.

Japanese translation guideline: The Japanese QAs are translated from the Vietnamese ones, with many entities and attributes in the sentences retained during the translation as possible. For transcribing Vietnamese proper nouns or other foreign words, the katakana syllabary is adopted. The polite form is utilized for writing translated questions and answers that contain verbs. In the case of complex Vietnamese questions and answers with multiple relevant information that may not be easily translated into continuous Japanese sentences, translators can use commas to split the sentences into smaller parts and then translate them subsequently. For example, the question ”What products does the woman wearing a helmet go to the store to buy?” can be translated to Japanese as ”ヘルメットをかぶった女性が店に買いに行って、どうの商品を買いますか?”, which literally means ”A woman wearing a helmet goes shopping at a store, what products does she buy?”.

After the human-translation process, we obtained 10,539 question-answer pairs in English and 11,562 question-answer pairs in Japanese.

4.3 Pre-processing and splitting

We normalize the Vietnamese and English QAs into lowercase. Latin characters in Japanese words, ”Tシャツ” (T-shirt) for instance, are also normalized similarly. After that slight pre-processing step, we merge all the QAs in all three languages into one dataset.

To prepare for the VLSP 2022 - EVJVQA Challenge, we split the dataset into training set, public test set, and private test set. We provide 3,763 images for the training set, while each test set comprises 558 images. Since no image is placed in multiple sets at once, we ensure that all QAs for a particular image only appear in the set to which that image belongs. The corresponding number of QAs in each language for each of the sets is shown in Table 3.

4.4 Statistics

As our dataset was constructed from three languages: Vietnamese, English, and Japanese, we conducted statistics to deeply observe the characteristic of each language as well as gain clear insights into the three languages.

To conduct statistics on Vietnamese, we use the word segmentation method from the VnCoreNLP Vu et al. (2018) as in Vietnamese, a word may have more than one token (for instance, ”cửa hàng tạp hóa” is formed from two Vietnamese words ”cửa hàng” and ”tạp hóa”, which is in turn formed from more than one token). For English QAs, we achieved tokens by splitting sentences using space. For Japanese QAs, like Vietnamese, Japanese uses hieroglyphs to form their word, hence each Japanese word may have more than one hieroglyph. We use the janome111https://github.com/mocobeta/janome library to perform word segmentation on Japanese QAs.

Table 4 indicate, with the same meaning, Japanese in general use more words to describe than Vietnamese and English. This implies another challenge for VQA method when tackling Japanese text beside the complexity of multilingualism in our dataset. Moreover, Vietnamese and English have the same distribution of length in terms of questions (according to Table 4), while English has shorter answers compared with those in Vietnamese.

Answers of the three languages share the same characteristic where the most appearance of length is two. This indicates humans while giving answers, prefer saying in short statements, and this behavior leads to the classification approach on the VQA dataset Teney et al. (2018). However, such short answers are not always the case in our dataset as the context of images taken in Vietnam is complicated because of the crowded scenes, traffic jams, or street stalls, and short answers are not enough to answer questions enquiring about complex scenes. Moreover, we aim to emphasize the language aspect of the VQA task, which means we want to guide the community to research and propose a system that can give answers flexibly and naturally as a human does, not the way of ”selecting” answers from a defined set as most approaches in the VQA dataset Antol et al. (2015); Teney et al. (2018). To this end, the answers given in our dataset are diverse in length and complicated in terms of level (word, phrase, or sentence level). Interestingly, while annotating answers, we found that giving a phrase or sentence as an answer is more fluent and human-like than giving only words or phrases as in the VQA dataset of Antol et al. (2015).

Another factor to consider while annotating the UIT-EVJVQA dataset is the language prior phenomenon Goyal et al. (2017). This is the phenomenon where the VQA methods try to learn patterns between questions and answers, such as questions starting with ”how many” usually go with the answer ”two”. As analyzed in Goyal et al. (2017), the language priors in VQA dataset is the result of the classification approach proposed for VQA task Teney et al. (2018) and cause the model to learn the way of recognizing answer based on the question rather than the way of how to make use of the image to answer the given question. Therefore while constructing the guideline to annotate the UIT-EVJVQA dataset, we propose to give answers using words, phrases, or sentences. In this way, we can first eliminate the traditional classification approach proposed for the VQA task in English as well as avoid the language priors in our dataset.

5 Systems and Results

The aim of the challenge is to evaluate the quality of the teams’ approaches to multilingual visual question-answering systems.

5.1 Baseline System

Following Changpinyo et al. Changpinyo et al. (2022c), we adopt transfer learning based on Vision Transformer (ViT) Dosovitskiy et al. (2020) and mBERT Devlin et al. (2019) for our baseline system. This work uses the training set for fine-tuning the pre-trained mBERT Devlin et al. (2019) model before generating answers. In addition, we use the ViT Dosovitskiy et al. (2020) to extract the visual features of images (Figure 7). Pre-trained ViT and mBERT models are initialized from HuggingFace checkpoints. We trained the baseline with a batch size of 64 and adapted the learning rate scheduler from Vaswani et al. Vaswani et al. (2017) to reduce the learning rate gradually after a number of iterations. The training process was interrupted automatically if the evaluation scores did not increase after 5 epochs.

5.2 Challenge Submission

The competition was hosted offline where the organizers provide the training set and public test set to the participant teams to evaluate and fine-tune their methods. When the competition came into the private test phase, the submission policy allows each participant team to submit up to 3 different methods during a submission time lasting 3 days. After the three-day private test phase, we evaluated their submitted results and obtain their F1 score as well as avg. BLEU score on the private test set. The final score of each participant team is the highest score among their submitted models on the private test set and their rank was indicated based on the F1 score ((BLUE as a secondary score when there is a tie).

5.3 Experimental Results

There are 8 participant teams submitted in total, and 5 of them have F1 scores higher than the baseline system.

6 Result Analysis

We mainly analyze the results of methods coming from participating teams based on the length of questions and answers in each language. To alleviate the result analysis, we define the four different ranges for the length of questions and answers. In particular, we define the short questions (respective, short answer) are questions whose length is shorter or equal to 5 tokens, medium questions (respective, medium answer) questions whose length is between 6 to 10 tokens, long questions (respective, long answer) are questions whose length is between 11 to 15 tokens, and finally, very long questions (respective, very long answer) are questions whose length is greater than 15 tokens. Tokens of questions and answers are defined in the same manner when we do statistics in Section 4.4. We sequentially report the results of the top-5 models in terms of scores and visualization in quantitative analysis and qualitative analysis, respectively, to support our statements as well as indicate what the research community should pay attention to when constructing the novel open-ended VQA dataset.

From Table 6, we have the top-5 models share the same characteristic on the English part of the UIT-EVJVQA dataset. Particularly, when we observe the results based on the question length, we can see that all submitted models have the same behavior when they give a higher performance on short questions and medium questions, while they yield a few drawbacks on long questions and very long questions. Turn the attention to the last four rows, which are the results based on answer length. We have different behavior. The top-5 models give better results on medium and long answers while yielding worse results on short and very long answers. Come from Figure 5, we can see that most English answers are short answers, which means models have more short answers to learning hence logically, they should have better performance on short answers rather than medium answers and long answers. To answer this weird insight, we showed all answers given by the top-5 models, and we found out that all top-5 models tend to give medium answers, even for questions having short answers. The most exciting thing here is how models give medium or lengthy answers: they repeat some tokens from the questions, and this is the primary way our annotators give medium and lengthy answers to questions. For questions with medium or long answers, answers from top-5 models mostly repeat some tokens from questions, and the gold medium or gold answers also repeat some tokens from questions. Thanks to these matched tokens with questions, F1 scores, and avg. BLEU scores of the answers from top-5 models are pretty high, while the crucial tokens, which determine whether or not the information in these answers is correct or not, usually have a length of 2 or 3 tokens; in case of totally wrong, they still do not affect the overall scores significantly. To gain a better understanding, we show some samples in the Appendix for an intuitive explanation.

Coming into the Vietnamese part of the UIT-EVJVQA dataset results, we have quite different behavior where most models perform better on medium and long questions. When we observe the length distribution of questions in Figure 4, most questions have a length range of around eight tokens, or most have medium length. While in English, most questions fell in the range of 5 tokens and seven tokens. Therefore the top-5 models have a good performance on short and medium questions in English, while they achieved better performance on medium and long answers in Vietnamese. For Vietnamese answers, as indicated in Table 7, top-5 models share the same performance as the English answers: they perform better on medium and long answers than on short and very long answers. We also provided some samples in Appendix to better demonstrate the method with English answers.

On the Japanese part of the UIT-EVJVQA dataset, the top-5 models have the same color on the English part when they give better results on medium and long questions. But unlike their performance in Vietnamese and English, where they achieved better scores for medium and long answers, the top-5 models have results as increase as the length of answers. From Figure 6, although the occurrence of short answers is the highest, the cumulative of medium and long answers are higher than the cumulative of short answers, which indicates the top-5 models not only have more medium and lengthy answers to learn but also tend to give medium or long answers to have the optimal loss on the training set. We also provided some samples in the Appendix for better demonstration.

Apart from the previous discussion, we can conclude on the UIT-EVJVQA dataset, most deep learning models tend to give lengthy answers to given questions with images, and the way they give lengthy answers is by repeatedly using some tokens of questions as starting point of the answers, and the main wrong parts of these answers are at the tokens indicating vital information to answer the questions such as objects, colors or side (see Appendix for more examples). Hence, to better understand how wrong the top-5 model gives predictions, we conduct analyses focusing on the usage of side, object, and color words in each language.

Side words: One of the most confusing attributes while inferring the description of objects is the use of side or direction. This is the case where an object is observed on the left or right of another object or appears on the left or right side of the scene. In English, side words are the words ”left” and ”right” in our dataset. While in Japanese, they are ”左” and ”右”. However, they are not simply ”trái” and ”phải” in Vietnamese, as ”trái” may not often be used to convey a side. Therefore, we broaden the set of side words as ”bên trái”, ”bên phải”, ”tay trái” (left-hand side) and ”tay phải” (right-hand side) according to the observation of the dataset. To measure how well each submitted model uses those side words in each language, we adopt the F1-score to calculate the proportion of match in terms of side words between the predictions and the gold answers. According to the results in Table 9, most models failed to indicate the side of objects while answering the questions.

Color words: As an attribute frequently appears in the QAs of our dataset, color is worth being observed as part of the inference performance of the models. Here we measure the F1-score of color word matches between predicted and gold answers for every team. The color words have been defined in the guideline in Section 4.2.1. However, there are some minor exceptions due to the crowdsourcing process.

We also visualize the color word distribution in the prediction of every team for each language. It is worth noting that in the Vietnamese QAs, there is a substantial quantity of ”xanh” color words which are unclear to be whether green or blue, but rather depend on the context of the visual scene. As we can see in Figure 8(a), Figure 8(b) and Figure 8(c), the ground truth color words are not evenly distributed because some colors such as white, black and red are used more frequently in the dataset, while the instances of brown, gray and purple are scant. This skewness is then emphasized through the overall inference of models. In Japanese, most submitted models clearly express bias as they intensively describe objects in white color, while less-appeared colors are poorly used. This kind of behavior is similar to the prior-language phenomenon pointed out in the previous work Goyal et al. (2017) as when being given questions about colors, top-5 models tend to use most-appearance colors despite the colors in images, and this phenomenon is also noticeable in other languages for some teams. For instance, the submitted model of team VL-UIT even ignores many of the less-appeared colors. As an attribute that is regularly used to describe an object, color also plays an important role in distinguishing between objects, which is an essential factor in visual question answering. Therefore, more efforts must be conducted to degrade the skewness in the color distribution in a way that helps the model infer better and more precisely describe objects out of images. We suggest future works should define a fixed range of colors and pay attention while asking and using colors to answer in order that we can ignore such prior-language phenomena on the VQA dataset.

Objects: To get the distribution of objects in the UIT-EVJVQA dataset, we used the POS method from the previous work Vu et al. (2018) and achieved objects by collecting tokens tagged as nouns. We used the same manner as investigating the behavior of top-5 models on color words but we can not find such language-prior phenomenon on objects. To find out the reason why most of the top-5 models failed in recognizing objects in images, we observed the results of pre-trained image models. However, most of the top-5 model’s used pre-trained image models that output grid-based features Jiang et al. (2020) such as ViT Dosovitskiy et al. (2020) or BEiT Bao et al. (2021), which means it is hard to visualize and interpret the results of those image models. Nevertheless, we can interpret results of other similar pre-trained image models such as Faster-RCNN Ren et al. (2015b) used ResNet101 He et al. (2016) as the backbone or Swin Transformer for object detection Liu et al. (2021) as these models were pre-trained on the ImageNet dataset Russakovsky et al. (2015) as well before being fine-tuned on various tasks, hence we used the outcomes of these two models on the training set of the UIT-EVJVQA dataset as an approximate way to investigate the reason of failure in recognizing an object in images of the top-5 models. As depicted in Figure 9, these two image models accurately detect objects that they were trained, but lots of common objects in Vietnam such as non la or fan (the first column of three images in Figure 9 or some traditional products available in most of the markets in Vietnam (the last column of three images in Figure 9) are not detected, and some of the detected objects in Figure 9 are not correct. This result implies the incorrect image understanding of pre-trained images models trained on images captured outside of Vietnam and indirectly affects the performance of VQA models (in case of using grid-based features, as region-based features Jiang et al. (2020) were achieved from grid-based features, e.g. from the backbone of Faster-RCNN, hence the failures indicated from region-based features indicates as well the failure in grid-based features). In conclusion available pre-trained image models are not relevant to scenes captured in Vietnam and the Computer Vision (CV) community in Vietnam should research and develop a better appropriate pre-trained image model, especially for images taken in Vietnam so that we can effectively tackle recent trending tasks where multi modeling task such as VQA is one of them.

7 Conclusion and Future Work

The VLSP2022-EVJVQA Challenge on multilingual image-based question answering has been organized at the VLSP 2022. Even though 57 teams had legally signed up to get the training dataset, only eight teams submitted their results. Because several teams enrolled for many challenges at the VLSP 2022, the other teams may not have enough time to explore VQA models. The highest performances are 0.4392 in F1-score and 0.4009 in BLUE on the private test set. The multilingual VQA systems proposed by the top 2 teams use ViT for the pre-trained vision model and mT5 for the pre-trained language model. EVJVQA is a challenging dataset including the training set, the development set (public test set), and the test set (private test set) that motivates NLP and CV researchers to further explore the multilingual models or systems in visual question answering.

To increase performance in multilingual visual question answering, we intend to increase the amount and quality of annotated questions in the future. In addition, we also make human-adversarial questions based on findings proposed by the research work Sheng et al. (2021).

Acknowledgements

We would like to thank the efforts of annotators who have contributed to the construction of a high-quality resource for the natural language processing research community. The VLSP 2022 was supported by organizations: Aimsoft, Bee, INT2, and DAGORAS, and educational organizations: VNU-HCM University of Information Technology, VNU University of Science, VNU University of Engineering and Technology, Hanoi University of Science and Technology, Vietnam Lexicography Centre, University of Science and Technology of Hanoi, ThuyLoi University, and VAST Institute of Information Technology. This work is partially supported by Vingroup Innovation Foundation (VINIF) in project code VINIF.2020.DA14.

Appendix A Appendix

In this section we provide extensively samples from top-5 models on the three languages for better demonstrating out observation in Section 6.