Parallel In-context Learning for Large Vision Language Models

TL;DR

This paper introduces Parallel-ICL, a method that enables large vision-language models to process long demonstration contexts efficiently by parallelizing inference and combining predictions, maintaining accuracy while reducing latency.

Contribution

The paper proposes a novel parallel inference algorithm for multi-modal in-context learning that significantly reduces inference time without sacrificing performance.

Findings

Parallel-ICL achieves comparable accuracy to full-context ICL.

It significantly reduces inference latency.

The method is effective across multiple vision-language tasks.

Abstract

Large vision-language models (LVLMs) employ multi-modal in-context learning (MM-ICL) to adapt to new tasks by leveraging demonstration examples. While increasing the number of demonstrations boosts performance, they incur significant inference latency due to the quadratic computational cost of Transformer attention with respect to the context length. To address this trade-off, we propose Parallel In-Context Learning (Parallel-ICL), a plug-and-play inference algorithm. Parallel-ICL partitions the long demonstration context into multiple shorter, manageable chunks. It processes these chunks in parallel and integrates their predictions at the logit level, using a weighted Product-of-Experts (PoE) ensemble to approximate the full-context output. Guided by ensemble learning theory, we introduce principled strategies for Parallel-ICL: (i) clustering-based context chunking to maximize inter-chunk diversity and (ii) similarity-based context compilation to weight predictions by query relevance. Extensive experiments on VQA, image captioning, and classification benchmarks demonstrate that Parallel-ICL achieves performance comparable to full-context MM-ICL, while significantly improving inference speed. Our work offers an effective solution to the accuracy-efficiency trade-off in MM-ICL, enabling dynamic task adaptation with substantially reduced inference overhead.