Bayesian-guided Label Mapping for Visual Reprogramming

TL;DR

This paper introduces Bayesian-guided Label Mapping (BLM), a probabilistic approach that improves visual reprogramming by capturing complex relationships between pretrained and downstream labels, outperforming existing methods.

Contribution

The paper proposes a novel Bayesian-guided label mapping method that constructs a probabilistic label relationship matrix, enhancing visual reprogramming performance.

Findings

BLM outperforms existing label mapping methods on various vision models.

The probabilistic approach provides better understanding of label relationships.

Experiments validate the effectiveness of BLM across different models.

Abstract

Visual reprogramming (VR) leverages the intrinsic capabilities of pretrained vision models by adapting their input or output interfaces to solve downstream tasks whose labels (i.e., downstream labels) might be totally different from the labels associated with the pretrained models (i.e., pretrained labels). When adapting the output interface, label mapping methods transform the pretrained labels to downstream labels by establishing a gradient-free one-to-one correspondence between the two sets of labels. However, in this paper, we reveal that one-to-one mappings may overlook the complex relationship between pretrained and downstream labels. Motivated by this observation, we propose a Bayesian-guided Label Mapping (BLM) method. BLM constructs an iteratively-updated probabilistic label mapping matrix, with each element quantifying a pairwise relationship between pretrained and downstream labels. The assignment of values to the constructed matrix is guided by Bayesian conditional probability, considering the joint distribution of the downstream labels and the labels predicted by the pretrained model on downstream samples. Experiments conducted on both pretrained vision models (e.g., ResNeXt) and vision-language models (e.g., CLIP) demonstrate the superior performance of BLM over existing label mapping methods. The success of BLM also offers a probabilistic lens through which to understand and analyze the effectiveness of VR. Our code is available at https://github.com/tmlr-group/BayesianLM.