Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models

TL;DR

This paper introduces Lever-LM, a tiny language model that configures effective in-context demonstration sequences to significantly enhance the in-context learning performance of large vision-language models across tasks like VQA and image captioning.

Contribution

The study proposes Lever-LM, a novel approach that captures internal statistical patterns to optimize demonstration sequences, improving LVLMs' performance in vision-language tasks.

Findings

Lever-LM improves ICL performance on VQA and captioning tasks.

Effective ICD sequences can be configured using Lever-LM.

Lever-LM captures statistical patterns for better model leveraging.

Abstract

As Archimedes famously said, ``Give me a lever long enough and a fulcrum on which to place it, and I shall move the world'', in this study, we propose to use a tiny Language Model (LM), \eg, a Transformer with 67M parameters, to lever much larger Vision-Language Models (LVLMs) with 9B parameters. Specifically, we use this tiny \textbf{Lever-LM} to configure effective in-context demonstration (ICD) sequences to improve the In-Context Learinng (ICL) performance of LVLMs. Previous studies show that diverse ICD configurations like the selection and ordering of the demonstrations heavily affect the ICL performance, highlighting the significance of configuring effective ICD sequences. Motivated by this and by re-considering the the process of configuring ICD sequence, we find this is a mirror process of human sentence composition and further assume that effective ICD configurations may contain internal statistical patterns that can be captured by Lever-LM. Then a dataset with effective ICD sequences is constructed to train Lever-LM. After training, given novel queries, new ICD sequences are configured by the trained Lever-LM to solve vision-language tasks through ICL. Experiments show that these ICD sequences can improve the ICL performance of two LVLMs compared with some strong baselines in Visual Question Answering and Image Captioning, validating that Lever-LM can really capture the statistical patterns for levering LVLMs. The code is available at \url{https://github.com/ForJadeForest/Lever-LM}.