Towards Global Optimal Visual In-Context Learning Prompt Selection

TL;DR

This paper introduces Partial2Global, a transformer-based framework for selecting the best in-context examples in visual learning tasks, significantly improving prompt selection and performance across multiple vision applications.

Contribution

The paper proposes a novel list-wise ranking approach with a consistency-aware aggregator to approximate the global optimal prompt in visual in-context learning, outperforming existing methods.

Findings

Partial2Global outperforms previous prompt selection methods.

It achieves state-of-the-art results in segmentation, detection, and colorization.

The framework demonstrates consistent improvement across diverse visual tasks.

Abstract

Visual In-Context Learning (VICL) is a prevailing way to transfer visual foundation models to new tasks by leveraging contextual information contained in in-context examples to enhance learning and prediction of query sample. The fundamental problem in VICL is how to select the best prompt to activate its power as much as possible, which is equivalent to the ranking problem to test the in-context behavior of each candidate in the alternative set and select the best one. To utilize more appropriate ranking metric and leverage more comprehensive information among the alternative set, we propose a novel in-context example selection framework to approximately identify the global optimal prompt, i.e. choosing the best performing in-context examples from all alternatives for each query sample. Our method, dubbed Partial2Global, adopts a transformer-based list-wise ranker to provide a more comprehensive comparison within several alternatives, and a consistency-aware ranking aggregator to generate globally consistent ranking. The effectiveness of Partial2Global is validated through experiments on foreground segmentation, single object detection and image colorization, demonstrating that Partial2Global selects consistently better in-context examples compared with other methods, and thus establish the new state-of-the-arts.