OAD-Promoter: Enhancing Zero-shot VQA using Large Language Models with Object Attribute Description

TL;DR

OAD-Promoter enhances zero-shot VQA with LLMs by reducing language bias and boosting out-of-distribution generalization through object-focused data augmentation and knowledge retrieval.

Contribution

It introduces a novel framework combining object-centric example generation, knowledge assistance, and specialized prompting to improve LLM-based VQA performance in zero-shot scenarios.

Findings

Achieves state-of-the-art results in zero-shot VQA tasks.

Significantly reduces language bias in LLM predictions.

Improves robustness to out-of-distribution questions.

Abstract

Large Language Models (LLMs) have become a crucial tool in Visual Question Answering (VQA) for handling knowledge-intensive questions in few-shot or zero-shot scenarios. However, their reliance on massive training datasets often causes them to inherit language biases during the acquisition of knowledge. This limitation imposes two key constraints on existing methods: (1) LLM predictions become less reliable due to bias exploitation, and (2) despite strong knowledge reasoning capabilities, LLMs still struggle with out-of-distribution (OOD) generalization. To address these issues, we propose Object Attribute Description Promoter (OAD-Promoter), a novel approach for enhancing LLM-based VQA by mitigating language bias and improving domain-shift robustness. OAD-Promoter comprises three components: the Object-concentrated Example Generation (OEG) module, the Memory Knowledge Assistance (MKA) module, and the OAD Prompt. The OEG module generates global captions and object-concentrated samples, jointly enhancing visual information input to the LLM and mitigating bias through complementary global and regional visual cues. The MKA module assists the LLM in handling OOD samples by retrieving relevant knowledge from stored examples to support questions from unseen domains. Finally, the OAD Prompt integrates the outputs of the preceding modules to optimize LLM inference. Experiments demonstrate that OAD-Promoter significantly improves the performance of LLM-based VQA methods in few-shot or zero-shot settings, achieving new state-of-the-art results.