Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey

TL;DR

This survey reviews recent advances in multimodal large language models for visual question answering, focusing on natural language understanding, knowledge reasoning, and multimodal fusion techniques.

Contribution

It provides a comprehensive overview of the development, models, knowledge reasoning, and evaluation methods in VQA using MLLMs, highlighting recent progress and future directions.

Findings

Recent models improve multimodal fusion accuracy

Knowledge reasoning enhances VQA performance

Evaluation metrics vary across datasets

Abstract

Visual Question Answering (VQA) is a challenge task that combines natural language processing and computer vision techniques and gradually becomes a benchmark test task in multimodal large language models (MLLMs). The goal of our survey is to provide an overview of the development of VQA and a detailed description of the latest models with high timeliness. This survey gives an up-to-date synthesis of natural language understanding of images and text, as well as the knowledge reasoning module based on image-question information on the core VQA tasks. In addition, we elaborate on recent advances in extracting and fusing modal information with vision-language pretraining models and multimodal large language models in VQA. We also exhaustively review the progress of knowledge reasoning in VQA by detailing the extraction of internal knowledge and the introduction of external knowledge. Finally, we present the datasets of VQA and different evaluation metrics and discuss possible directions for future work.