Robust Nearest Neighbour Retrieval Using Targeted Manifold Manipulation

TL;DR

The paper introduces TMM-NN, a novel nearest neighbour retrieval method that uses targeted perturbations to assess sample responsiveness, improving robustness and semantic relevance over traditional distance metrics.

Contribution

It presents a new retrieval approach based on targeted manifold manipulation, utilizing a lightweight trigger patch and weak backdoor to enhance robustness and relevance.

Findings

Outperforms traditional metrics under noise conditions.

Effective in diverse retrieval tasks.

Robust against adversarial perturbations.

Abstract

Nearest-neighbour retrieval is central to classification and explainable-AI pipelines, but current practice relies on hand-tuning feature layers and distance metrics. We propose Targeted Manifold Manipulation-Nearest Neighbour (TMM-NN), which reconceptualises retrieval by assessing how readily each sample can be nudged into a designated region of the feature manifold; neighbourhoods are defined by a sample's responsiveness to a targeted perturbation rather than absolute geometric distance. TMM-NN implements this through a lightweight, query-specific trigger patch. The patch is added to the query image, and the network is weakly ``backdoored'' so that any input with the patch is steered toward a dummy class. Images similar to the query need only a slight shift and are classified as the dummy class with high probability, while dissimilar ones are less affected. By ranking candidates by this confidence, TMM-NN retrieves the most semantically related neighbours. Robustness analysis and benchmark experiments confirm this trigger-based ranking outperforms traditional metrics under noise and across diverse tasks.