Joint Semantic Token Selection and Prompt Optimization for Interpretable Prompt Learning

TL;DR

This paper introduces IPL, a hybrid framework for prompt learning that combines discrete semantic token selection with continuous prompt optimization, enhancing interpretability and accuracy in vision-language models.

Contribution

It proposes a novel alternating optimization approach for semantic token selection and prompt tuning, improving interpretability without sacrificing adaptability.

Findings

IPL improves interpretability and accuracy across multiple benchmarks.

The framework is plug-and-play and compatible with existing prompt learning methods.

Extensive experiments validate the effectiveness and scalability of IPL.

Abstract

Vision-language models such as CLIP achieve strong visual-textual alignment, but often suffer from overfitting and limited interpretability when adapted through continuous prompt learning. While discrete prompt optimization improves interpretability, it usually depends on large external models, leading to high computational costs and limited scalability. In this paper, we propose Interpretable Prompt Learning (IPL), a hybrid framework that alternates between discrete semantic token selection and continuous prompt optimization. Specifically, IPL formulates semantic token selection as an approximate submodular optimization problem, encouraging tokens that are both human-understandable and semantically diverse. It further adopts an alternating optimization strategy to integrate discrete token selection with continuous prompt tuning, improving interpretability while preserving adaptability to downstream tasks. Our framework is plug-and-play, allowing seamless integration with existing prompt learning methods. Extensive experiments on multiple benchmarks show that IPL consistently improves both interpretability and accuracy across five representative prompt learning methods, providing an effective and scalable extension to existing frameworks.