T2T-VICL: Unlocking the Boundaries of Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

TL;DR

This paper introduces T2T-VICL, a novel framework that enables cross-task visual in-context learning using vision-language models, by generating task-specific prompts and combining perceptual reasoning with traditional metrics, achieving top-tier results.

Contribution

The paper presents the first cross-task VICL dataset and a new inference framework that enhances VLMs' ability to perform across diverse visual tasks.

Findings

Achieves top-tier results in twelve cross-task scenarios.

Demonstrates the effectiveness of implicit text prompts for cross-task VICL.

Unlocks new potential for VLMs in multi-task visual understanding.

Abstract

In large language models (LLM), in-context learning (ICL) refers to performing new tasks by conditioning on small demonstrations provided in the input context. Recent advances in visual in-context learning (VICL) demonstrate promising capabilities for solving downstream tasks by unified vision-language models (VLMs). When the visual prompt and the target images originate from different visual tasks, can VLMs still enable VICL? In the paper, we propose a fully collaborative pipeline, i.e. T2T-VICL, for VLMs to investigate the potential of cross-task VICL. Fundamentally, we design a mechanism to generate and select text prompts that best implicitly describe the differences between two distinct low-level vision tasks, and construct the first cross-task VICL dataset. Building upon this, we propose a novel inference framework that combines perceptual score-based reasoning with traditional evaluation metrics to perform cross-task VICL. Our approach achieves top-tier results across twelve cross-task scenarios and second-tier performance in nine additional scenarios, unlocking the boundaries of cross-task VICL within VLMs.