Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions

TL;DR

This paper introduces a new robustness evaluation method for VQA models using semantically related basic questions as noise, and shows that in-context learning with these questions can improve accuracy.

Contribution

It proposes a novel robustness measure R_score, a ranking method for basic questions via LASSO, and demonstrates the effectiveness of in-context learning with basic questions.

Findings

The proposed method effectively evaluates VQA model robustness.

In-context learning with basic questions enhances model accuracy.

The new robustness measure R_score provides a standardized evaluation metric.

Abstract

Deep neural networks have been critical in the task of Visual Question Answering (VQA), with research traditionally focused on improving model accuracy. Recently, however, there has been a trend towards evaluating the robustness of these models against adversarial attacks. This involves assessing the accuracy of VQA models under increasing levels of noise in the input, which can target either the image or the proposed query question, dubbed the main question. However, there is currently a lack of proper analysis of this aspect of VQA. This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models. It is hypothesized that as the similarity of a basic question to the main question decreases, the level of noise increases. To generate a reasonable noise level for a given main question, a pool of basic questions is ranked based on their similarity to the main question, and this ranking problem is cast as a LASSO optimization problem. Additionally, this work proposes a novel robustness measure, R_score, and two basic question datasets to standardize the analysis of VQA model robustness. The experimental results demonstrate that the proposed evaluation method effectively analyzes the robustness of VQA models. Moreover, the experiments show that in-context learning with a chain of basic questions can enhance model accuracy.