A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021

TL;DR

This paper introduces a transformer-based cross-modal fusion model enhanced with adversarial training for visual question answering, achieving state-of-the-art results on the VQAv2 dataset.

Contribution

It combines vision-language pre-trained models with adversarial training and implementation tricks to improve robustness and performance in VQA tasks.

Findings

Achieved 76.72% accuracy on VQAv2 test-std set.

Enhanced model robustness through adversarial training.

Improved results with specific implementation techniques.

Abstract

In this paper, inspired by the successes of visionlanguage pre-trained models and the benefits from training with adversarial attacks, we present a novel transformerbased cross-modal fusion modeling by incorporating the both notions for VQA challenge 2021. Specifically, the proposed model is on top of the architecture of VinVL model [19], and the adversarial training strategy [4] is applied to make the model robust and generalized. Moreover, two implementation tricks are also used in our system to obtain better results. The experiments demonstrate that the novel framework can achieve 76.72% on VQAv2 test-std set.