Variational Visual Question Answering for Uncertainty-Aware Selective Prediction

TL;DR

This paper introduces Variational VQA, a Bayesian approach that enhances the reliability and safety of large vision-language models in visual question answering by improving calibration and selective prediction.

Contribution

It demonstrates the effectiveness of variational Bayes for selective prediction in VQA, outperforming traditional methods and proposing a new risk-averse selector for better uncertainty handling.

Findings

Variational VQA improves calibration and selective prediction accuracy.

A single posterior sample can outperform models trained with AdamW.

The risk-averse selector outperforms standard sample averaging.

Abstract

Despite remarkable progress in recent years, Vision Language Models (VLMs) remain prone to overconfidence and hallucinations on tasks such as Visual Question Answering (VQA) and Visual Reasoning. Bayesian methods can potentially improve reliability by helping models predict selectively, that is, models respond only when they are sufficiently confident. Unfortunately, such approaches can be costly and ineffective for large models, and there exists little evidence to show otherwise for multimodal applications. Here, we show for the first time the effectiveness and competitive edge of variational Bayes for selective prediction in VQA. We build on recent advances in variational methods for deep learning and propose an extension called "Variational VQA". This method improves calibration and yields significant gains for selective prediction on VQA and Visual Reasoning, particularly when the error tolerance is low ($\leq 1\%$). Often, just one posterior sample yields more reliable answers than those given by models trained with AdamW. In addition, we propose a new risk-averse selector that outperforms standard sample averaging by considering the variance of predictions. Overall, we present compelling evidence that variational learning is a viable option to make large VLMs safer and more trustworthy.