Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning

TL;DR

This paper introduces a reinforcement learning framework for multi-modal demonstration selection in large vision-language models, significantly improving few-shot learning performance on VQA tasks by optimizing demonstration policies.

Contribution

It proposes a novel exploration-exploitation reinforcement learning approach that adaptively fuses multi-modal information and selects demonstrations, surpassing heuristic methods.

Findings

Outperforms existing methods on four VQA datasets

Enhances generalization of few-shot LVLMs

Demonstrates effective autonomous policy refinement

Abstract

In-context learning (ICL), a predominant trend in instruction learning, aims at enhancing the performance of large language models by providing clear task guidance and examples, improving their capability in task understanding and execution. This paper investigates ICL on Large Vision-Language Models (LVLMs) and explores the policies of multi-modal demonstration selection. Existing research efforts in ICL face significant challenges: First, they rely on pre-defined demonstrations or heuristic selecting strategies based on human intuition, which are usually inadequate for covering diverse task requirements, leading to sub-optimal solutions; Second, individually selecting each demonstration fails in modeling the interactions between them, resulting in information redundancy. Unlike these prevailing efforts, we propose a new exploration-exploitation reinforcement learning framework, which explores policies to fuse multi-modal information and adaptively select adequate demonstrations as an integrated whole. The framework allows LVLMs to optimize themselves by continually refining their demonstrations through self-exploration, enabling the ability to autonomously identify and generate the most effective selection policies for in-context learning. Experimental results verify the superior performance of our approach on four Visual Question-Answering (VQA) datasets, demonstrating its effectiveness in enhancing the generalization capability of few-shot LVLMs.