IOTA: Corrective Knowledge-Guided Prompt Learning via Black-White Box Framework

TL;DR

IOTA introduces a novel framework combining knowledge-driven and data-driven modules to improve pre-trained model adaptation for image classification, outperforming state-of-the-art methods in few-shot and varied difficulty settings.

Contribution

The paper proposes a black-white box prompt learning framework that effectively integrates corrective knowledge with data-driven learning for better downstream task adaptation.

Findings

Outperforms state-of-the-art methods on 12 image classification benchmarks.

Effective in few-shot and easy-to-hard adaptation scenarios.

Utilizes interpretable prompts derived from corrective knowledge.

Abstract

Recently, adapting pre-trained models to downstream tasks has attracted increasing interest. Previous Parameter-Efficient-Tuning (PET) methods regard the pre-trained model as an opaque Black Box model, relying purely on data-driven optimization and underutilizing their inherent prior knowledge. This oversight limits the models' potential for effective downstream task adaptation. To address these issues, we propose a novel black-whIte bOx prompT leArning framework (IOTA), which integrates a data-driven Black Box module with a knowledge-driven White Box module for downstream task adaptation. Specifically, the White Box module derives corrective knowledge by contrasting the wrong predictions with the right cognition. This knowledge is verbalized into interpretable human prompts and leveraged through a corrective knowledge-guided prompt selection strategy to guide the Black Box module toward more accurate predictions. By jointly leveraging knowledge- and data-driven learning signals, IOTA achieves effective downstream task adaptation. Experimental results on 12 image classification benchmarks under few-shot and easy-to-hard adaptation settings demonstrate the effectiveness of corrective knowledge and the superiority of our method over state-of-the-art methods.