Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering

TL;DR

This paper introduces a meta-learning approach that distills task-relevant visual features into soft prompts, enabling large multimodal models to adapt effectively to new visual question answering tasks with minimal data and outperforming traditional in-context learning.

Contribution

It proposes a novel meta-learning method with soft prompt distillation and an attention-mapper for improved few-shot VQA adaptation in large multimodal models.

Findings

Achieves 21.2% improvement over in-context learning.

Enhances adaptation by 7.7% over parameter-efficient fine-tuning methods.

Effective in low-data regimes with just a few gradient steps.

Abstract

Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to perform new visual question answering (VQA) tasks with minimal supervision. However, ICL performance, especially in smaller LMMs, does not always improve monotonically when increasing the number of examples. We hypothesize that this happens because the LMM is overwhelmed by extraneous information in the image embeddings that is irrelevant to the downstream task. To address this, we propose a meta-learning approach that induces few-shot capabilities in LMMs through a fixed set of soft prompts distilled from task-relevant visual features, which are adapted at test time using a small number of examples. We facilitate this distillation through an attention-mapper module that can be easily integrated with any LMM architecture and is jointly learned with soft prompts. Evaluation on the VL-ICL Bench shows that our method successfully achieves task adaptation in low-data regimes with just a few gradient steps, outperforming ICL by 21.2%. Comparisons with parameter-efficient finetuning methods demonstrate that meta-learning further enhances this adaptation by 7.7% for various VQA tasks.