Targeted Mismatch Adversarial Attack: Query with a Flower to Retrieve the Tower

TL;DR

This paper presents a targeted mismatch adversarial attack method that generates deceptive images to conceal user queries in visual search engines, successfully fooling partially unknown retrieval systems while maintaining similar retrieval results.

Contribution

The paper introduces a novel targeted mismatch attack approach for deep retrieval systems, including loss functions for unknown system components, and demonstrates effectiveness on standard benchmarks.

Findings

Successful attacks on partially unknown retrieval systems.

Effective concealment of user queries with minimal retrieval difference.

Robustness of attack across different system configurations.

Abstract

Access to online visual search engines implies sharing of private user content - the query images. We introduce the concept of targeted mismatch attack for deep learning based retrieval systems to generate an adversarial image to conceal the query image. The generated image looks nothing like the user intended query, but leads to identical or very similar retrieval results. Transferring attacks to fully unseen networks is challenging. We show successful attacks to partially unknown systems, by designing various loss functions for the adversarial image construction. These include loss functions, for example, for unknown global pooling operation or unknown input resolution by the retrieval system. We evaluate the attacks on standard retrieval benchmarks and compare the results retrieved with the original and adversarial image.