NAU-QMUL: Utilizing BERT and CLIP for Multi-modal AI-Generated Image Detection

TL;DR

This paper presents a multi-modal, multi-task model utilizing BERT and CLIP for detecting AI-generated images and identifying their sources, demonstrating competitive performance in a major AI-generated image detection competition.

Contribution

It introduces a novel multi-modal architecture with cross-modal feature fusion and pseudo-labeling data augmentation for improved AI-generated image detection.

Findings

Achieved fifth place in the CT2 competition for AI-generated image detection.

Attained F1 scores of 83.16% and 48.88% in Tasks A and B.

Demonstrated the effectiveness of multi-modal fusion and data augmentation strategies.

Abstract

With the aim of detecting AI-generated images and identifying the specific models responsible for their generation, we propose a multi-modal multi-task model. The model leverages pre-trained BERT and CLIP Vision encoders for text and image feature extraction, respectively, and employs cross-modal feature fusion with a tailored multi-task loss function. Additionally, a pseudo-labeling-based data augmentation strategy was utilized to expand the training dataset with high-confidence samples. The model achieved fifth place in both Tasks A and B of the `CT2: AI-Generated Image Detection' competition, with F1 scores of 83.16\% and 48.88\%, respectively. These findings highlight the effectiveness of the proposed architecture and its potential for advancing AI-generated content detection in real-world scenarios. The source code for our method is published on https://github.com/xxxxxxxxy/AIGeneratedImageDetection.