Cross-modal Adversarial Reprogramming

TL;DR

This paper explores adversarial reprogramming to repurpose pre-trained image classifiers for NLP and sequence tasks by designing efficient programs that map text to images, achieving competitive results without retraining.

Contribution

It extends adversarial reprogramming to cross-modal tasks, enabling image classifiers to perform NLP and sequence classification without modifying their architecture or parameters.

Findings

Reprogrammed image classifiers perform well on NLP benchmarks.

Efficient adversarial programs enable cross-modal task adaptation.

No retraining of the original model is required.

Abstract

With the abundance of large-scale deep learning models, it has become possible to repurpose pre-trained networks for new tasks. Recent works on adversarial reprogramming have shown that it is possible to repurpose neural networks for alternate tasks without modifying the network architecture or parameters. However these works only consider original and target tasks within the same data domain. In this work, we broaden the scope of adversarial reprogramming beyond the data modality of the original task. We analyze the feasibility of adversarially repurposing image classification neural networks for Natural Language Processing (NLP) and other sequence classification tasks. We design an efficient adversarial program that maps a sequence of discrete tokens into an image which can be classified to the desired class by an image classification model. We demonstrate that by using highly efficient adversarial programs, we can reprogram image classifiers to achieve competitive performance on a variety of text and sequence classification benchmarks without retraining the network.