FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering

TL;DR

This paper introduces FRAMES-VQA, a comprehensive benchmark for evaluating the robustness of fine-tuning methods in visual question answering systems across diverse multi-modal distribution shifts, including in-distribution and out-of-distribution scenarios.

Contribution

It proposes a new benchmark for multi-modal VQA robustness evaluation, categorizes datasets into various shift types, and provides detailed analyses of distribution shifts and modality importance.

Findings

Existing fine-tuning methods vary in robustness across shifts.

Distribution shifts can be quantified using Mahalanobis distance.

Interactions between uni- and multi-modal shifts influence model performance.

Abstract

Visual question answering (VQA) systems face significant challenges when adapting to real-world data shifts, especially in multi-modal contexts. While robust fine-tuning strategies are essential for maintaining performance across in-distribution (ID) and out-of-distribution (OOD) scenarios, current evaluation settings are primarily unimodal or particular to some types of OOD, offering limited insight into the complexities of multi-modal contexts. In this work, we propose a new benchmark FRAMES-VQA (Fine-Tuning Robustness across Multi-Modal Shifts in VQA) for evaluating robust fine-tuning for VQA tasks. We utilize ten existing VQA benchmarks, including VQAv2, IV-VQA, VQA-CP, OK-VQA and others, and categorize them into ID, near and far OOD datasets covering uni-modal, multi-modal and adversarial distribution shifts. We first conduct a comprehensive comparison of existing robust fine-tuning methods. We then quantify the distribution shifts by calculating the Mahalanobis distance using uni-modal and multi-modal embeddings extracted from various models. Further, we perform an extensive analysis to explore the interactions between uni- and multi-modal shifts as well as modality importance for ID and OOD samples. These analyses offer valuable guidance on developing more robust fine-tuning methods to handle multi-modal distribution shifts. The code is available at https://github.com/chengyuehuang511/FRAMES-VQA .