Controlled Caption Generation for Images Through Adversarial Attacks
Nayyer Aafaq, Naveed Akhtar, Wei Liu, Mubarak Shah, Ajmal Mian

TL;DR
This paper introduces a GAN-based method to generate adversarial images that manipulate CNN features to produce targeted or keyword-specific incorrect captions in image captioning systems, revealing vulnerabilities.
Contribution
It presents a novel adversarial attack approach focusing on CNN hidden layers, providing new insights into the robustness of vision-language models.
Findings
Effective adversarial images can fool captioning systems
The attack demonstrates high transferability across models
It enables targeted and keyword-specific caption manipulation
Abstract
Deep learning is found to be vulnerable to adversarial examples. However, its adversarial susceptibility in image caption generation is under-explored. We study adversarial examples for vision and language models, which typically adopt an encoder-decoder framework consisting of two major components: a Convolutional Neural Network (i.e., CNN) for image feature extraction and a Recurrent Neural Network (RNN) for caption generation. In particular, we investigate attacks on the visual encoder's hidden layer that is fed to the subsequent recurrent network. The existing methods either attack the classification layer of the visual encoder or they back-propagate the gradients from the language model. In contrast, we propose a GAN-based algorithm for crafting adversarial examples for neural image captioning that mimics the internal representation of the CNN such that the resulting deep features…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Adversarial Robustness in Machine Learning · Domain Adaptation and Few-Shot Learning
