OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning

TL;DR

This paper introduces OpenworldAUC, a unified evaluation metric for open-world prompt tuning that assesses both detection and classification, and proposes Gated Mixture-of-Prompts (GMoP) for optimized performance, demonstrating state-of-the-art results across benchmarks.

Contribution

The paper proposes OpenworldAUC as a new unified metric and GMoP as an optimization method for open-world prompt tuning, addressing evaluation and training challenges.

Findings

OpenworldAUC effectively evaluates detection and classification jointly.

GMoP achieves state-of-the-art performance on 15 benchmarks.

The method is insensitive to varying base/new sample ratios.

Abstract

Prompt tuning adapts Vision-Language Models like CLIP to open-world tasks with minimal training costs. In this direction, one typical paradigm evaluates model performance separately on known classes (i.e., base domain) and unseen classes (i.e., new domain). However, real-world scenarios require models to handle inputs without prior domain knowledge. This practical challenge has spurred the development of open-world prompt tuning, which demands a unified evaluation of two stages: 1) detecting whether an input belongs to the base or new domain (P1), and 2) classifying the sample into its correct class (P2). What's more, as domain distributions are generally unknown, a proper metric should be insensitive to varying base/new sample ratios (P3). However, we find that current metrics, including HM, overall accuracy, and AUROC, fail to satisfy these three properties simultaneously. To bridge this gap, we propose OpenworldAUC, a unified metric that jointly assesses detection and classification through pairwise instance comparisons. To optimize OpenworldAUC effectively, we introduce Gated Mixture-of-Prompts (GMoP), which employs domain-specific prompts and a gating mechanism to dynamically balance detection and classification. Theoretical guarantees ensure generalization of GMoP under practical conditions. Experiments on 15 benchmarks in open-world scenarios show GMoP achieves SOTA performance on OpenworldAUC and other metrics. We release the code at https://github.com/huacong/OpenworldAUC