Towards Counterfactual Image Manipulation via CLIP

TL;DR

This paper introduces a novel text-driven counterfactual image editing method using CLIP, enabling realistic and accurate modifications of images based on counterfactual concepts by exploiting semantic directions and embedding mapping.

Contribution

The work proposes a new contrastive loss and embedding mapping scheme to improve counterfactual image editing with CLIP, addressing challenges of semantic guidance and editing precision.

Findings

Achieves accurate counterfactual image editing driven by target texts.

Produces realistic edits that align with desired counterfactual concepts.

Demonstrates effectiveness across diverse semantic editing scenarios.

Abstract

Leveraging StyleGAN's expressivity and its disentangled latent codes, existing methods can achieve realistic editing of different visual attributes such as age and gender of facial images. An intriguing yet challenging problem arises: Can generative models achieve counterfactual editing against their learnt priors? Due to the lack of counterfactual samples in natural datasets, we investigate this problem in a text-driven manner with Contrastive-Language-Image-Pretraining (CLIP), which can offer rich semantic knowledge even for various counterfactual concepts. Different from in-domain manipulation, counterfactual manipulation requires more comprehensive exploitation of semantic knowledge encapsulated in CLIP as well as more delicate handling of editing directions for avoiding being stuck in local minimum or undesired editing. To this end, we design a novel contrastive loss that exploits predefined CLIP-space directions to guide the editing toward desired directions from different perspectives. In addition, we design a simple yet effective scheme that explicitly maps CLIP embeddings (of target text) to the latent space and fuses them with latent codes for effective latent code optimization and accurate editing. Extensive experiments show that our design achieves accurate and realistic editing while driving by target texts with various counterfactual concepts.