Enhancing Visual In-Context Learning by Multi-Faceted Fusion

TL;DR

This paper introduces a multi-faceted fusion framework for Visual In-Context Learning that leverages collaborative integration of multiple prompts, significantly improving model performance across various visual tasks.

Contribution

It proposes a novel multi-combination collaborative fusion approach and the MULTI-VQGAN architecture to better utilize diverse contextual prompts in VICL.

Findings

Enhanced performance on foreground segmentation, object detection, and image colorization.

Superior cross-task generalization and robustness.

More accurate predictions compared to existing prompt fusion methods.

Abstract

Visual In-Context Learning (VICL) has emerged as a powerful paradigm, enabling models to perform novel visual tasks by learning from in-context examples. The dominant "retrieve-then-prompt" approach typically relies on selecting the single best visual prompt, a practice that often discards valuable contextual information from other suitable candidates. While recent work has explored fusing the top-K prompts into a single, enhanced representation, this still simply collapses multiple rich signals into one, limiting the model's reasoning capability. We argue that a more multi-faceted, collaborative fusion is required to unlock the full potential of these diverse contexts. To address this limitation, we introduce a novel framework that moves beyond single-prompt fusion towards an multi-combination collaborative fusion. Instead of collapsing multiple prompts into one, our method generates three contextual representation branches, each formed by integrating information from different combinations of top-quality prompts. These complementary guidance signals are then fed into proposed MULTI-VQGAN architecture, which is designed to jointly interpret and utilize collaborative information from multiple sources. Extensive experiments on diverse tasks, including foreground segmentation, single-object detection, and image colorization, highlight its strong cross-task generalization, effective contextual fusion, and ability to produce more robust and accurate predictions than existing methods.