Towards Identity-Aware Cross-Modal Retrieval: a Dataset and a Baseline

TL;DR

This paper introduces a new dataset and baseline model for identity-aware cross-modal retrieval, enabling better recognition of specific individuals in images based on natural language queries, especially for long-tail identities.

Contribution

The paper presents a novel dataset, COCO-PFS, and a baseline architecture, Id-CLIP, for improving identity-aware cross-modal retrieval tasks.

Findings

Id-CLIP achieves competitive retrieval performance after fine-tuning.

The dataset enables training and evaluation for long-tail identity recognition.

Experiments demonstrate the effectiveness of targeted fine-tuning for this task.

Abstract

Recent advancements in deep learning have significantly enhanced content-based retrieval methods, notably through models like CLIP that map images and texts into a shared embedding space. However, these methods often struggle with domain-specific entities and long-tail concepts absent from their training data, particularly in identifying specific individuals. In this paper, we explore the task of identity-aware cross-modal retrieval, which aims to retrieve images of persons in specific contexts based on natural language queries. This task is critical in various scenarios, such as for searching and browsing personalized video collections or large audio-visual archives maintained by national broadcasters. We introduce a novel dataset, COCO Person FaceSwap (COCO-PFS), derived from the widely used COCO dataset and enriched with deepfake-generated faces from VGGFace2. This dataset addresses the lack of large-scale datasets needed for training and evaluating models for this task. Our experiments assess the performance of different CLIP variations repurposed for this task, including our architecture, Identity-aware CLIP (Id-CLIP), which achieves competitive retrieval performance through targeted fine-tuning. Our contributions lay the groundwork for more robust cross-modal retrieval systems capable of recognizing long-tail identities and contextual nuances. Data and code are available at https://github.com/mesnico/IdCLIP.