Cheap-fake Detection with LLM using Prompt Engineering

TL;DR

This paper introduces a novel approach for detecting out-of-context media, specifically cheapfakes, by leveraging prompt-engineered features from GPT-3.5 to enhance a coherence assessment model, significantly improving detection accuracy.

Contribution

It presents a new method that uses prompt engineering with GPT-3.5 as a feature extractor to improve out-of-context media detection within the COSMOS framework.

Findings

Enhanced detection performance with GPT-3.5 features

Effective integration of prompt-engineered features into baseline model

Potential applications in NLP, image captioning, and text-to-image synthesis

Abstract

The misuse of real photographs with conflicting image captions in news items is an example of the out-of-context (OOC) misuse of media. In order to detect OOC media, individuals must determine the accuracy of the statement and evaluate whether the triplet (~\textit{i.e.}, the image and two captions) relates to the same event. This paper presents a novel learnable approach for detecting OOC media in ICME'23 Grand Challenge on Detecting Cheapfakes. The proposed method is based on the COSMOS structure, which assesses the coherence between an image and captions, as well as between two captions. We enhance the baseline algorithm by incorporating a Large Language Model (LLM), GPT3.5, as a feature extractor. Specifically, we propose an innovative approach to feature extraction utilizing prompt engineering to develop a robust and reliable feature extractor with GPT3.5 model. The proposed method captures the correlation between two captions and effectively integrates this module into the COSMOS baseline model, which allows for a deeper understanding of the relationship between captions. By incorporating this module, we demonstrate the potential for significant improvements in cheap-fakes detection performance. The proposed methodology holds promising implications for various applications such as natural language processing, image captioning, and text-to-image synthesis. Docker for submission is available at https://hub.docker.com/repository/docker/mulns/ acmmmcheapfakes.