Composite Concept Extraction through Backdooring

TL;DR

This paper presents CoCE, a novel zero-shot method for extracting composite concepts by repurposing backdoor attack techniques and contrastive learning, enabling the identification of complex concepts from simple examples.

Contribution

It introduces a new approach that leverages backdoor attack mechanisms and contrastive learning to extract composite concepts in a zero-shot setting, which was not previously possible.

Findings

Effective in extracting composite concepts across multiple datasets

Utilizes backdoor techniques for strategic distortion in feature space

Demonstrates robustness and applicability in various scenarios

Abstract

Learning composite concepts, such as \textquotedbl red car\textquotedbl , from individual examples -- like a white car representing the concept of \textquotedbl car\textquotedbl{} and a red strawberry representing the concept of \textquotedbl red\textquotedbl -- is inherently challenging. This paper introduces a novel method called Composite Concept Extractor (CoCE), which leverages techniques from traditional backdoor attacks to learn these composite concepts in a zero-shot setting, requiring only examples of individual concepts. By repurposing the trigger-based model backdooring mechanism, we create a strategic distortion in the manifold of the target object (e.g., \textquotedbl car\textquotedbl ) induced by example objects with the target property (e.g., \textquotedbl red\textquotedbl ) from objects \textquotedbl red strawberry\textquotedbl , ensuring the distortion selectively affects the target objects with the target property. Contrastive learning is then employed to further refine this distortion, and a method is formulated for detecting objects that are influenced by the distortion. Extensive experiments with in-depth analysis across different datasets demonstrate the utility and applicability of our proposed approach.